tracdap.api

Module Contents

Classes

DataReadRequest

Request to read a dataset stored in the TRAC platform.

DataReadResponse

Response to a request to read a dataset stored in the TRAC platform.

DataWriteRequest

Request to create of update a dataset stored in the TRAC platform.

FileReadRequest

Request to read a FILE, i.e. retrieve its content from the platform.

FileReadResponse

Response to a request to read a FILE, i.e. retrieve its content from the platform.

FileWriteRequest

Request to create or update a FILE and store content to the platform.

JobRequest

JobStatus

JobStatusRequest

ListTenantsRequest

Request object for the listTenants() API call.

ListTenantsResponse

Response object for the listTenants() API call.

MetadataBatchRequest

Request to read a batch of objects from the TRAC metadata store.

MetadataBatchResponse

Response to reading a batch of objects from the TRAC metadata store.

MetadataGetRequest

Request to get an object from the TRAC metadata store. This is available as an alternative to MetadataGetRequest for use in REST-style GET calls.

MetadataReadRequest

Request to read an object from the TRAC metadata store. An alternative REST-style GET call is available using MetadataGetRequest.

MetadataSearchRequest

Request for a search against the TRAC metadata store.

MetadataSearchResponse

Response to a search against the TRAC metadata store.

MetadataWriteBatchRequest

Request to create or update a batch of objects in the TRAC metadata store.

MetadataWriteBatchResponse

Response to creating or updating a batch of objects in the TRAC metadata store.

MetadataWriteRequest

Request to create or update an object in the TRAC metadata store.

PlatformInfoRequest

Request object for the platformInfo() API call.

PlatformInfoResponse

Response object for the platformInfo() API call.

TracDataApi

Public API for creating, updating, reading and querying primary data stored in the TRAC platform.

TracMetadataApi

Public API for creating, updating, reading and searching for objects in the TRAC metadata store.

TracOrchestratorApi

class tracdap.api.DataReadRequest

Request to read a dataset stored in the TRAC platform.

See also

readDataset

format

Format that the dataset will be returned in.

The format must be the mime type of a supported data format. Data will be encoded in this format before it is sent back.

Type:

str

selector

Selector for the dataset being read

Type:

metadata.TagSelector

tenant

Tenant code for the requested operation, always required.

Type:

str

class tracdap.api.DataReadResponse

Response to a request to read a dataset stored in the TRAC platform.

See also

readDataset

content

Content of the dataset, encoded in the requested format.

The first message in the response stream will always contain an empty content buffer. Subsequent messages will provided the content of the dataset as a series of chunks.

Type:

bytes

schema

Schema of the dataset being returned.

The schema is provided in the first message of the response stream. TRAC will always provide a full schema, regardless of whether the dataset is using am external schema or an embedded one.

Type:

Optional[metadata.SchemaDefinition]

class tracdap.api.DataWriteRequest

Request to create of update a dataset stored in the TRAC platform.

content

The content of the dataset, encoded as per the “format” field.

For client streaming requests, each message contains a chunk of the encoded dataset. Empty (i.e. zero-length) chunks can be included in the stream and will be ignored. For unary requests (createSmallDataset, updateSmallDataset) the encoded content of the dataset must be included in the request as a single blob.

Type:

bytes

format

Format that will be used for uploading data.

This field tells TRAC what format the data will be sent in, it must be the mime type of a supported data format. Data provided in the “content” field can then be encoded in this format.

The “format” field does not affect how TRAC stores data. Incoming data is decoded and translated into a format recognised by the underlying data platform.

Type:

str

priorVersion

Prior object/tag version to use for update operations.

This field should be omitted when creating a new dataset.

Type:

Optional[metadata.TagSelector]

schema

Specify the schema with an explicit schema definition.

If an explicit schema is supplied TRAC will save the schema as part of the DATA object, creating an embedded schema this individual dataset. All future versions of the dataset must also use an embedded schema which can either be the same schema or a compatible one, with compatibility following the rules of schema versioning (fields can be added but not removed or materially altered).

TRAC will validate incoming data to make sure it is conformant with the schema.

Type:

Optional[metadata.SchemaDefinition]

schemaId

Specify the schema using an external schema definition.

If a schema ID is supplied, it must be a valid tag selector for an existing SCHEMA object. All future versions of the dataset must use either the same schema ID or a later version of the same schema object.

TRAC will look up the external schema and validate incoming data to make sure it is conformant.

Type:

Optional[metadata.TagSelector]

tagUpdates

Tag update operations to be applied.

Tag updates are applied in exactly the same way as for metadata write requests.

Type:

List[metadata.TagUpdate]

tenant

Tenant code for the requested operation, always required.

Type:

str

class tracdap.api.FileReadRequest

Request to read a FILE, i.e. retrieve its content from the platform.

See also

readFile

selector

Selector for the FILE being read

Type:

metadata.TagSelector

tenant

Tenant code for the requested operation, always required

Type:

str

class tracdap.api.FileReadResponse

Response to a request to read a FILE, i.e. retrieve its content from the platform.

See also

readFile

content

A chunk of the file content

The file content will be sent as a stream of chunks (byte buffers), with one chunk in each message. Empty chunks may be included at any point in the response stream and should be ignored. In particular, TRAC guarantees that the chunk in the first message will always be an empty chunk.

Type:

bytes

fileDefinition

Definition of the FILE being read

Type:

Optional[metadata.FileDefinition]

class tracdap.api.FileWriteRequest

Request to create or update a FILE and store content to the platform.

content

A chunk of the file content

The file content should be sent as a stream of chunks (byte buffers), with one chunk in each message. Empty chunks can be included at any point in the request stream and will be ignored.

Type:

bytes

mimeType

Mime type of the file being saved, always required

Must be a valid mime type. For update requests, the mime type must match exactly with the mime type supplied when the file was originally created. Attempting to change the mime type in an update will result in an error status of FAILED_PRECONDITION.

Mime type is stored in the file definition and in the trac_file_mime_type attribute.

Type:

str

name

File name of the file being saved, always required

Must be a valid file name with no path component (i.e. the name may not contain slashes).

If the name contains a period character, the file extension will be set as the portion of the name following the last period. Otherwise the file extension will be blank.

For update requests the name may change but the extension must stay the same. E.g. a file can be created as “my_file_v1.txt” and updated as “my_file_v2.txt”, but updating with the name as “my_file_v1.doc” would not be allowed. Attempting to change the extension in an update will result in an error status of FAILED_PRECONDITION.

Fle name and extension are stored in the file definition, and in the trac_file_name and trac_file_extension attributes.

Type:

str

priorVersion

Prior object/tag version to use for update operations

This field should be omitted when creating a new file.

Type:

Optional[metadata.TagSelector]

size

Size of the file being saved, if known in advance

When a size is supplied, TRAC will check the size against the number of bytes stored. If the stored file size does not match the supplied value, the error will be reported with an error status of DATA_LOSS. When no size is supplied the check cannot be performed.

File size is stored in the created file definition and in the trac_file_size attribute.

Type:

Optional[int]

tagUpdates

Tag update operations to be applied

Tag updates are applied in exactly the same way as for metadata write requests.

Type:

List[metadata.TagUpdate]

tenant

Tenant code for the requested operation, always required

Type:

str

class tracdap.api.JobRequest
job

job

Type:

metadata.JobDefinition

jobAttrs

jobAttrs

Type:

List[metadata.TagUpdate]

tenant

tenant

Type:

str

class tracdap.api.JobStatus
jobId

jobId

Type:

metadata.TagHeader

statusCode

statusCode

Type:

metadata.JobStatusCode

statusMessage

statusMessage

Type:

str

class tracdap.api.JobStatusRequest
selector

selector

Type:

metadata.TagSelector

tenant

tenant

Type:

str

class tracdap.api.ListTenantsRequest

Request object for the listTenants() API call.

Currently the API call does not require any parameters.

class tracdap.api.ListTenantsResponse

Response object for the listTenants() API call.

tenants
  • A list of TenantInfo objects, one for each tenant the current user has access to. *

Type:

List[metadata.TenantInfo]

class tracdap.api.MetadataBatchRequest

Request to read a batch of objects from the TRAC metadata store.

See also

TracMetadataApi

selector

selector

Type:

List[metadata.TagSelector]

tenant

tenant

Type:

str

class tracdap.api.MetadataBatchResponse

Response to reading a batch of objects from the TRAC metadata store.

See also

TracMetadataApi

tag

tag

Type:

List[metadata.Tag]

class tracdap.api.MetadataGetRequest

Request to get an object from the TRAC metadata store. This is available as an alternative to MetadataGetRequest for use in REST-style GET calls.

See also

TracMetadataApi

objectId

objectId

Type:

str

objectType

objectType

Type:

metadata.ObjectType

objectVersion

objectVersion

Type:

Optional[int]

tagVersion

tagVersion

Type:

Optional[int]

tenant

tenant

Type:

str

class tracdap.api.MetadataReadRequest

Request to read an object from the TRAC metadata store. An alternative REST-style GET call is available using MetadataGetRequest.

See also

TracMetadataApi

selector

selector

Type:

metadata.TagSelector

tenant

tenant

Type:

str

class tracdap.api.MetadataSearchRequest

Request for a search against the TRAC metadata store.

See also

TracMetadataApi

searchParams

searchParams

Type:

metadata.SearchParameters

tenant

tenant

Type:

str

class tracdap.api.MetadataSearchResponse

Response to a search against the TRAC metadata store.

See also

TracMetadataApi

searchResult

searchResult

Type:

List[metadata.Tag]

class tracdap.api.MetadataWriteBatchRequest

Request to create or update a batch of objects in the TRAC metadata store.

See also

TracMetadataApi

batchAttrs

These attrs will be appended to tagUpdates in each request.

Type:

List[metadata.TagUpdate]

requests

Requests to be realized in batch.

Tenant in requests must be null.

Type:

List[MetadataWriteRequest]

tenant

Tenant code for the requested operation, always required.

Type:

str

class tracdap.api.MetadataWriteBatchResponse

Response to creating or updating a batch of objects in the TRAC metadata store.

See also

TracMetadataApi

headers

Headers of the written tags.

In the same order as in MetadataWriteBatchRequest.

Type:

List[metadata.TagHeader]

class tracdap.api.MetadataWriteRequest

Request to create or update an object in the TRAC metadata store.

See also

TracMetadataApi

definition

New object definition to save.

A definition is required to create or update an object, for tag updates the definition field should be left blank.

This field should be omitted for tag update operations.

Type:

Optional[metadata.ObjectDefinition]

objectType

Object type for the requested operation, always required.

Type:

metadata.ObjectType

priorVersion

Prior object/tag version to use for update operations.

This field should be omitted when creating a new object.

Type:

Optional[metadata.TagSelector]

tagUpdates

Tag update operations to be applied.

The updates are applied in order and must all succeeded in order for the request to be successful. If the API call fails for any reason, none of the requested tag updates will be applied.

Type:

List[metadata.TagUpdate]

tenant

Tenant code for the requested operation, always required.

Type:

str

class tracdap.api.PlatformInfoRequest

Request object for the platformInfo() API call.

Currently the API call does not require any parameters.

class tracdap.api.PlatformInfoResponse

Response object for the platformInfo() API call.

Contains some general information about the TRAC platform.

deploymentInfo

Additional information about the current TRAC environment.

The particular information relevant to a deployment of TRAC can vary between organizations. Useful information might include: region, business division, brand etc. This information is optional and may not be present for smaller deployments.

Type:

Dict[str, str]

environment
  • Name of the current TRAC environment.

Type:

str

production
  • Flag indicating whether the current environment is a production environment or not. *

Type:

bool

tracVersion
  • Version string for the version of TRAC currently deployed in the platform. *

Type:

str

class tracdap.api.TracDataApi

Public API for creating, updating, reading and querying primary data stored in the TRAC platform.

The TRAC data API provides a standard mechanism for client applications to store and access data in the TRAC platform. Calls are translated into the underlying storage mechanisms, using push-down operations where possible for efficient queries on large datasets.

The data API includes format translation, so data can be uploaded and retrieved in any supported format. The back-end storage format is controlled by the platform. For example if a user uploads a CSV file TRAC will convert it to the default storage format (by default Arrow IPC file format). Later a web application might ask for that data in JSON format and TRAC would again perform the conversion. The platform uses Apache Arrow as an intermediate representation that other formats are converted from and to.

The data API uses streaming operations to support transfer of large datasets and can be used for both user-facing client applications and system-to-system integration. For regular, high-volume data transfers there are other options for integration, including data import jobs and direct back-end integration of the underlying storage technologies. These options can be faster and reduce storage requirements, at the expense of tighter coupling between systems. A compromise is to use direct integration and import jobs for a small number of critical feeds containing a high volume of data, and use the data API for client access and for integration of secondary and/or low-volume systems.

createDataset(request)

Create a new dataset, supplying the schema and content as a data stream

This method creates a new dataset and a corresponding DATA object in the TRAC metadata store. Once a dataset is created it can be used as an input into a model run, it can also be read and queried using the data API. Data can be supplied in any format supported by the platform.

Although large datasets can be uploaded using this call, data import jobs are normally used to bring in large volumes of data from external systems.

The request must specify a schema for the dataset, incoming data will be verified against the schema. Schemas can be specified using either:

  • A full schema definition - if a full schema is supplied, it will be embedded with the dataset and used for this dataset only

  • A schema ID - a tag selector for an existing SCHEMA object, which may be shared by multiple datasets

The “format” parameter describes the format used to upload data. For example, to upload a CSV file the format would be set to “text/csv” and the file content can be uploaded directly, or to upload the output of an editor grid in a web client the format can be set to “text/json” to upload a JSON representation of the editor contents. TRAC will apply format conversion before the data is processed and stored.

Tag updates can be supplied to tag the newly created dataset, they behave exactly the same as tag updates in the createObject() call of TracMetadataApi.

This is a client streaming method. The first message in the request stream must contain all the request fields and metadata, including a schema specifier. After the first message all metadata fields should be omitted. Subsequent messages should contain the content of the dataset as a series of chunks, encoded as per the “format” field of the first message. Clients may choose whether or not to include a chunk of data in the first message and empty (i.e. zero-length) chunks are permitted at any point in the stream.

This method returns the header of the newly created DATA object. Error conditions include: Invalid request, unknown tenant, schema not found (if an external schema ID is used), format not supported, data does not match schema, corrupt or invalid data stream. Storage errors may also be reported if there is a problem communicating with the underlying storage technology. In the event of an error, TRAC will do its best to clean up any partially-written data in the storage layer.

Parameters:

request (DataWriteRequest) –

Return type:

tracdap.metadata.TagHeader

createFile(request)

Upload a new file into TRAC, sending the content as a data stream

Calling this method will create a new FILE object in the metadata store. Tag updates can be supplied when creating a FILE, they will be passed on to the metadata service. The semantics for tag updates are identical to the createObject() method in TracMetadataApi.

This is a client streaming method. The first message in the request stream must contain all the request fields and required metadata. The second and subsequent messages should contain the content of the file as a series of chunks (byte buffers). (other fields that are set after the first message will be ignored). Empty chunks can be included at any point in the stream and will be ignored. Clients may choose to include the first chunk in the first message along with the request metadata, or to put an empty chunk in the first message and start streaming content in the second message. For very small files, it is possible to put the entire content in one chunk in the first message, so there is only a single message in the stream. All of these approaches are supported.

Clients may specify the size of the file being created. When a size is supplied, TRAC will check the size against the number of bytes stored. If the stored file size does not match the supplied value, the error will be reported with an error status of DATA_LOSS. When no size is supplied the check cannot be performed.

The method returns the header of the newly created FILE object. Error conditions include: Invalid request, unknown tenant and validation failure and data loss (if the number of bytes stored does not match the number specified in the request). Storage errors may also be reported if there is a problem communicating with the underlying storage technology. In the event of an error, TRAC will do its best to clean up any partially-written data in the storage layer.

Parameters:

request (FileWriteRequest) –

Return type:

tracdap.metadata.TagHeader

createSmallDataset(request)

Create a new dataset, supplying the schema and content as a single blob

This method creates a new dataset and a corresponding DATA object in the TRAC metadata store. Once a dataset is created it can be used as an input into a model run, it can also be read and queried using the data API. Data can be supplied in any format supported by the platform.

The request must specify a schema for the dataset, incoming data will be verified against the schema. Schemas can be specified using either:

  • A full schema definition - if a full schema is supplied, it will be embedded with the dataset and used for this dataset only

  • A schema ID - a tag selector for an existing SCHEMA object, which may be shared by multiple datasets

The “format” parameter describes the format used to upload data. For example, to upload a CSV file the format would be set to “text/csv” and the file content can be uploaded directly, or to upload the output of an editor grid in a web client the format can be set to “text/json” to upload a JSON representation of the editor contents. TRAC will apply format conversion before the data is processed and stored.

Tag updates can be supplied to tag the newly created dataset, they behave exactly the same as tag updates in the createObject() call of TracMetadataApi.

This is a unary call, all the request fields and metadata (including schema specifier) and dataset content encoded as per the “format” field are supplied in a single message. It is intended for working with small datasets and for use in environments where client streaming is not available (particularly in gRPC-Web clients).

This method returns the header of the newly created DATA object. Error conditions include: Invalid request, unknown tenant, schema not found (if an external schema ID is used), format not supported, data does not match schema, corrupt or invalid data stream. Storage errors may also be reported if there is a problem communicating with the underlying storage technology. In the event of an error, TRAC will do its best to clean up any partially-written data in the storage layer.

Parameters:

request (DataWriteRequest) –

Return type:

tracdap.metadata.TagHeader

createSmallFile(request)

Upload a new file into TRAC, sending the content as a single blob

Calling this method will create a new FILE object in the metadata store. Tag updates can be supplied when creating a FILE, they will be passed on to the metadata service. The semantics for tag updates are identical to the createObject() method in TracMetadataApi.

This is a unary method. The request must contain all the relevant fields and the entire content of the file in a single message. Errors may occur if the file is too large to fit in a single message frame.

Clients may specify the size of the file being created. When a size is supplied, TRAC will check the size against the number of bytes stored. If the stored file size does not match the supplied value, the error will be reported with an error status of DATA_LOSS. When no size is supplied the check cannot be performed.

The method returns the header of the newly created FILE object. Error conditions include: Invalid request, unknown tenant and validation failure, file too large and data loss (if the number of bytes stored does not match the number specified in the request). Storage errors may also be reported if there is a problem communicating with the underlying storage technology. In the event of an error, TRAC will do its best to clean up any partially-written data in the storage layer.

Parameters:

request (FileWriteRequest) –

Return type:

tracdap.metadata.TagHeader

readDataset(request)

Read an existing dataset, returning the content as a data stream

This method reads the contents of an existing dataset and returns it in the requested format, along with a copy of the data schema. Data can be requested in any format supported by the platform.

The request uses a regular TagSelector to indicate which dataset and version to read. The format parameter is a mime type and must be a supported data format.

This is a server streaming method. The first message in the response stream will contain a schema definition for the dataset (this may come from an embedded schema or an external schema object). The second and subsequent messages will deliver the content of the dataset in the requested format. TRAC guarantees that the first message will always contain an empty chunk of content, which can be safely ignored.

Error conditions include: Invalid request, unknown tenant, object not found, format not supported. Storage errors may also be reported if there is a problem communicating with the underlying storage technology.

Parameters:

request (DataReadRequest) –

Return type:

DataReadResponse

readFile(request)

Download a file that has been stored in TRAC and return it as a data stream

The request uses a regular TagSelector to indicate which file to read. The semantics of the request are identical to the readObject() method in TracMetadataApi.

This is a server streaming method. The first message in the response stream will contain the response metadata (i.e. the file definition). The second and subsequent messages will deliver the content of the file as a stream of chunks (byte buffers). Empty chunks may be included at any point in the stream and should be ignored. In particular, TRAC guarantees that the chunk in the first message will always be an empty chunk. Clients are free to ignore this chunk, for example if they have a separate function for processing the first message in the response stream. Alternatively clients may process the empty chunk in the firs message in the same way as any other chunk. Both approaches are supported.

Error conditions include: Invalid request, unknown tenant, unknown object ID, object type does not match ID, unknown object version, unknown tag version. Storage errors may also be reported if there is a problem communicating with the underlying storage technology.

Parameters:

request (FileReadRequest) –

Return type:

FileReadResponse

readSmallDataset(request)

Read an existing dataset, returning the content as a single blob

This method reads the contents of an existing dataset and returns it in the requested format, along with a copy of the data schema. Data can be requested in any format supported by the platform.

The request uses a regular TagSelector to indicate which dataset and version to read. The format parameter is a mime type and must be a supported data format.

This is a unary call, both the schema and the content of the dataset are returned in a single response message. The content of the dataset will be encoded in the requested format. Errors may occur if the content of the dataset is too large to fit in a single message frame.

Error conditions include: Invalid request, unknown tenant, object not found, format not supported. Storage errors may also be reported if there is a problem communicating with the underlying storage technology.

Parameters:

request (DataReadRequest) –

Return type:

DataReadResponse

readSmallFile(request)

Download a file that has been stored in TRAC and return it as a single blob

The request uses a regular TagSelector to indicate which file to read. The semantics of the request are identical to the readObject() method in TracMetadataApi.

This is a unary method, the response will contain the file definition and the whole content of the file in a single message. Errors may occur if the file is too large to fit in a single message frame.

Error conditions include: Invalid request, unknown tenant, unknown object ID, object type does not match ID, unknown object version, unknown tag version, file too large. Storage errors may also be reported if there is a problem communicating with the underlying storage technology.

Parameters:

request (FileReadRequest) –

Return type:

FileReadResponse

updateDataset(request)

Update an existing dataset, supplying the schema and content as a data stream

This method updates an existing dataset and the corresponding DATA object in the TRAC metadata store. As per the TRAC immutability guarantee, the original version of the dataset is not altered. After an update, both the original version and the new version are available to use as inputs into a model runs and to read and query using the data API. Data can be supplied in any format supported by the platform.

Although large datasets can be uploaded using this call, data import jobs are normally used to bring in large volumes of data from external systems.

To update a dataset, the priorVersion field must indicate the dataset being updated. Only the latest version of a dataset can be updated.

The request must specify a schema for the new version of the dataset, incoming data will be verified against the schema. The new schema must be compatible with the schema of the previous version. Schemas can be specified using either:

  • A full schema definition - Datasets created using an embedded schema must supply a full schema for all subsequent versions and each schema version must be compatible with the version before. Fields may be added, but not removed or altered.

  • A schema ID - Datasets created using an external schema must use the same external schema ID for all subsequent versions. It is permitted for later versions of a dataset to use later versions of the external schema, but not earlier versions.

The “format” parameter describes the format used to upload data. For example, to upload a CSV file the format would be set to “text/csv” and the file content can be uploaded directly, or to upload the output of an editor grid in a web client the format can be set to “text/json” to upload a JSON representation of the editor contents. It is not necessary for different versions of the same dataset to be uploaded using the same format. TRAC will apply format conversion before the data is processed and stored.

Tag updates can be supplied to tag the new version of the dataset, they behave exactly the same as tag updates in the updateObject() call of TracMetadataApi.

This is a client streaming method. The first message in the request stream must contain all the request fields and metadata, including a schema specifier. After the first message all metadata fields should be omitted. Subsequent messages should contain the content of the dataset as a series of chunks, encoded as per the “format” field of the first message. Clients may choose whether or not to include a chunk of data in the first message and empty (i.e. zero-length) chunks are permitted at any point in the stream.

This method returns the header of the version of the DATA object. Error conditions include: Invalid request, unknown tenant, schema not found (if an external schema ID is used), schema version not compatible, format not supported, data does not match schema, corrupt or invalid data stream. Storage errors may also be reported if there is a problem communicating with the underlying storage technology. In the event of an error, TRAC will do its best to clean up any partially-written data in the storage layer.

Parameters:

request (DataWriteRequest) –

Return type:

tracdap.metadata.TagHeader

updateFile(request)

Upload a new version of an existing file into TRAC, sending the content as a data stream

Calling this method will update the relevant FILE object in the metadata store. The latest version of the FILE must be supplied in the priorVersion field of the request. For example if the latest version of a FILE object is version 2, the priorVersion field should refer to version 2 and TRAC will create version 3 as a result of the update call. The metadata and content of prior versions remain unaltered. The file name may be changed between versions, but the extension and mime type must stay the same. Tag updates can be supplied when updating a FILE, they will be passed on to the metadata service> The semantics for tag updates are identical to the updateObject() method in TracMetadataApi.

This is a client streaming method. The first message in the request stream must contain all the request fields and required metadata. The second and subsequent messages should contain the content of the file as a series of byte buffers. (other fields that are set after the first message will be ignored). Empty chunks can be included at any point in the stream and will be ignored. Clients may choose to include the first chunk in the first message along with the request metadata, or to put an empty chunk in the first message and start streaming content in the second message. For very small files, it is possible to put the entire content in one chunk in the first message, so there is only a single message in the stream. All of these approaches are supported.

Clients may specify the size of the file being updated. When a size is supplied, TRAC will check the size against the number of bytes stored. If the stored file size does not match the supplied value, the error will be reported with an error status of DATA_LOSS. When no size is supplied the check cannot be performed.

The call returns the header for the new version of the FILE object. Error conditions include: Invalid request, unknown tenant, validation failure, failed preconditions (e.g. extension and mime type changes) and data loss (if the number of bytes stored does not match the number specified in the request). Storage errors may also be reported if there is a problem communicating with the underlying storage technology. In the event of an error, TRAC will do its best to clean up any partially-written data in the storage layer.

Parameters:

request (FileWriteRequest) –

Return type:

tracdap.metadata.TagHeader

updateSmallDataset(request)

Update an existing dataset, supplying the schema and content as a single blob

This method updates an existing dataset and the corresponding DATA object in the TRAC metadata store. As per the TRAC immutability guarantee, the original version of the dataset is not altered. After an update, both the original version and the new version are available to use as inputs into a model runs and to read and query using the data API. Data can be supplied in any format supported by the platform.

To update a dataset, the priorVersion field must indicate the dataset being updated. Only the latest version of a dataset can be updated.

The request must specify a schema for the new version of the dataset, incoming data will be verified against the schema. The new schema must be compatible with the schema of the previous version. Schemas can be specified using either:

  • A full schema definition - Datasets created using an embedded schema must supply a full schema for all subsequent versions and each schema version must be compatible with the version before. Fields may be added, but not removed or altered.

  • A schema ID - Datasets created using an external schema must use the same external schema ID for all subsequent versions. It is permitted for later versions of a dataset to use later versions of the external schema, but not earlier versions.

The “format” parameter describes the format used to upload data. For example, to upload a CSV file the format would be set to “text/csv” and the file content can be uploaded directly, or to upload the output of an editor grid in a web client the format can be set to “text/json” to upload a JSON representation of the editor contents. It is not necessary for different versions of the same dataset to be uploaded using the same format. TRAC will apply format conversion before the data is processed and stored.

Tag updates can be supplied to tag the new version of the dataset, they behave exactly the same as tag updates in the updateObject() call of TracMetadataApi.

This is a unary call, all the request fields and metadata (including schema specifier) and dataset content encoded as per the “format” field are supplied in a single message. It is intended for working with small datasets and for use in environments where client streaming is not available (particularly in gRPC-Web clients).

This method returns the header of the version of the DATA object. Error conditions include: Invalid request, unknown tenant, schema not found (if an external schema ID is used), schema version not compatible, format not supported, data does not match schema, corrupt or invalid data stream. Storage errors may also be reported if there is a problem communicating with the underlying storage technology. In the event of an error, TRAC will do its best to clean up any partially-written data in the storage layer.

Parameters:

request (DataWriteRequest) –

Return type:

tracdap.metadata.TagHeader

updateSmallFile(request)

Upload a new version of an existing file into TRAC, sending the content as a single blob

Calling this method will update the relevant FILE object in the metadata store. The latest version of the FILE must be supplied in the priorVersion field of the request. For example if the latest version of a FILE object is version 2, the priorVersion field should refer to version 2 and TRAC will create version 3 as a result of the update call. The metadata and content of prior versions remain unaltered. The file name may be changed between versions, but the extension and mime type must stay the same. Tag updates can be supplied when updating a FILE, they will be passed on to the metadata service> The semantics for tag updates are identical to the updateObject() method in TracMetadataApi.

This is a unary call. The request must contain all the relevant fields and the entire content of the file in a single message. Errors may occur if the file is too large to fit in a single message frame.

Clients may specify the size of the file being updated. When a size is supplied, TRAC will check the size against the number of bytes stored. If the stored file size does not match the supplied value, the error will be reported with an error status of DATA_LOSS. When no size is supplied the check cannot be performed.

The call returns the header for the new version of the FILE object. Error conditions include: Invalid request, unknown tenant, validation failure, failed preconditions (e.g. extension and mime type changes) file too large and data loss (if the number of bytes stored does not match the number specified in the request). Storage errors may also be reported if there is a problem communicating with the underlying storage technology. In the event of an error, TRAC will do its best to clean up any partially-written data in the storage layer.

Parameters:

request (FileWriteRequest) –

Return type:

tracdap.metadata.TagHeader

class tracdap.api.TracMetadataApi

Public API for creating, updating, reading and searching for objects in the TRAC metadata store.

The TRAC metadata API is expressed in terms of Objects and Tags. Objects are structural items (nouns and verbs) such as data, models and jobs that describe resources and operations on the TRAC platform. Tags contain descriptive and informational data and provide a means to classify, catalogue and control objects. Tags can also be used to describe the structure of applications built on the TRAC platform. For more information on the TRAC metadata model, refer to the documentation for ObjectDefinition and Tag.

The write API allows objects to be created directly, by supplying an object definition and a set of tags. These definitions and tags are validated, assigned an ID and version (in a tag header) then saved as they are. Only certain types of objects can be created / updated directly via the public API (currently FLOW and CUSTOM objects). Other types of objects are created by TRAC as a result of operations on the platform, for example a model import job will create a model definition.

Creating a new version of an object is very similar to creating a new object, in that the entire metadata definition must be supplied. TRAC will perform validation to make sure the new definition is a valid version update from the previous version - the semantics of versioning depend on the particular object type.

The metadata write API also provides a means for updating tags (i.e. creating a new tag version); tag updates are allowed for all object types via the public API. The current API requires a whole new tag to be supplied, this will be replaced in a future version with an API that supplies attributes changes as a set of add/modify/remove operations.

The read API is for reading metadata from the metadata store when the IDs of the metadata items are known. The easiest way to read metadata is using TagSelectors, which may be present in other metadata items (e.g. job definitions record tag selectors to indicate which versions of resources were used in a job) or can be easily constructed from an object ID or tag header. HTTP GET calls are also available for REST clients (REST clients can still use selectors with POST requests).

Metadata searches are performed using tag attributes, as well as an object type and a few top-level parameters to handle versioning and temporality. See the SearchParameters object for a more detailed description. The result of a search call is a list of matching tags, which are always arranged with the most recent tags first. A search that matches no results will return an empty list.

This API is a multi-tenant API. For gRPC requests every request includes a tenant code, for REST requests the tenant code is the first element of every path.

createObject(request)

Create a new object in the TRAC metadata store.

To create an object, supply a metadata write request with the tenant code, object type and a definition for the item you want to save. The tag should contain all the attributes you want to set and the object definition, without an object header. TRAC will create a header when it assigns and ID for the new object. Only FLOW and CUSTOM objects can be created by this API call.

Validation is performed on new objects before they are saved. Objects can fail validation because they are semantically invalid (e.g. missing required fields or inconsistencies within the object definition) or due to consistency errors (referring to other metadata items that don’t exist or don’t meet certain requirements).

NOTE: Validation is only partially implemented in the current release.

The call returns an ID response to indicate the ID of the newly created object, as well as the object and tag version (which will always be 1).

Error conditions include: Invalid request, unknown tenant, object type does not match the supplied definition, validation failure.

Parameters:

request (MetadataWriteRequest) –

Return type:

tracdap.metadata.TagHeader

createObjectBatch(request)

Create new objects in batch in the TRAC metadata store.

To create an object, supply a metadata write request with the tenant code, object type and a definition for the item you want to save. The tag should contain all the attributes you want to set and the object definition, without an object header. TRAC will create a header when it assigns and ID for the new object. Only FLOW and CUSTOM objects can be created by this API call.

Validation is performed on new objects before they are saved. Objects can fail validation because they are semantically invalid (e.g. missing required fields or inconsistencies within the object definition) or due to consistency errors (referring to other metadata items that don’t exist or don’t meet certain requirements).

NOTE: Validation is only partially implemented in the current release.

The call returns an ID responses to indicate the IDs of the newly created objects, as well as the objects and tag versions (which will always be 1).

Error conditions include: Invalid request, unknown tenant, object type does not match the supplied definition, validation failure.

Parameters:

request (MetadataWriteBatchRequest) –

Return type:

MetadataWriteBatchResponse

getLatestObject(request)

Get the latest version of a metadata object using an HTTP GET request.

This call is intended for use by REST clients to provide TRAC metadata objects as RESTFUL resources. For gRPC clients it is recommended to use readObject instead and supply a TagSelector.

See also

readObject

To use this call, supply a tenant code, object type and ID. The call will return the latest version of the object with its latest tag, including the full object definition.

Error conditions include: Invalid request, unknown tenant, unknown object ID, object type does not match ID.

Parameters:

request (MetadataGetRequest) –

Return type:

tracdap.metadata.Tag

getLatestTag(request)

Get the latest tag for a metadata object using an HTTP GET request.

This call is intended for use by REST clients to provide TRAC metadata objects as RESTFUL resources. For gRPC clients it is recommended to use readObject instead and supply a TagSelector.

See also

readObject

To use this call, supply a tenant code, object type, ID and version. The call will return the latest tag for the specified version of the object, including the full object definition.

Error conditions include: Invalid request, unknown tenant, unknown object ID, object type does not match ID, unknown object version.

Parameters:

request (MetadataGetRequest) –

Return type:

tracdap.metadata.Tag

getObject(request)

Get a single metadata object using an HTTP GET request.

This call is intended for use by REST clients to provide TRAC metadata objects as RESTFUL resources. For gRPC clients it is recommended to use readObject instead and supply a TagSelector.

See also

readObject

To use this call, supply a tenant code, object type, ID, object version and tag version. The call will return a tag for the exact object version and tag version specified, including the full object definition.

Error conditions include: Invalid request, unknown tenant, unknown object ID, object type does not match ID, unknown object version, unknown tag version.

Parameters:

request (MetadataGetRequest) –

Return type:

tracdap.metadata.Tag

listTenants(request)

Get a list of tenants that the current user has access to.

Information returned includes the available tenant codes, as well as human readable descriptions. Client tools that support multiple tenants and tenant switching will need to fetch this list.

Only tenants accessible to the current user are returned.

Parameters:

request (ListTenantsRequest) –

Return type:

ListTenantsResponse

platformInfo(request)

Get some general information about the TRAC platform.

Information returned includes what environment you are connected to and what version of TRAC is running. Useful for displaying in “about” boxes in a UI or client tool, and for submitting error reports.

Parameters:

request (PlatformInfoRequest) –

Return type:

PlatformInfoResponse

readBatch(request)

Read multiple objects from the TRAC metadata store using tag selectors.

Tag selectors are used throughout the TRAC metadata structures and API calls to refer to objects. For example, a job definition includes selectors to define the data sets and models that were selected for the job. Those selectors can be passed directly into this call.

Tag selectors can refer to object and tag versions either using explicit version numbers, with as-of times or by referring to the current latest version. Combinations are also possible (e.g. object version x, latest tag).

This call takes multiple selectors and returns multiple Tags in the same order, including their object definitions. Selectors for different object types and with different criteria for selecting versions can be mixed freely in a single request. Error conditions include: Invalid request, unknown tenant, unknown object ID, object type does not match ID, unknown object version, unknown tag version. If there is an error for any of the selectors the whole request will fail.

See also

TagSelector

Parameters:

request (MetadataBatchRequest) –

Return type:

MetadataBatchResponse

readObject(request)

Read a single object from the TRAC metadata store using a tag selector.

Tag selectors are used throughout the TRAC metadata structures and API calls to refer to objects. For example, a job definition includes selectors to define the data sets and models that were selected for the job. Those selectors can be passed directly into this call.

Tag selectors can refer to object and tag versions either using explicit version numbers, with as-of times or by referring to the current latest version. Combinations are also possible (e.g. object version x, latest tag).

This call takes a single selector and returns a single Tag, including the object definition. Error conditions include: Invalid request, unknown tenant, unknown object ID, object type does not match ID, unknown object version, unknown tag version.

See also

TagSelector

Parameters:

request (MetadataReadRequest) –

Return type:

tracdap.metadata.Tag

search(request)

Perform a search against the TRAC metadata store.

Searches are always for a specific tenant and object type. For details of the available search parameters, see SearchParameters.

In gRPC calls, a MetadataSearchRequest specifies the tenant and includes the SearchParameters, where object type and the search expression are set. For REST calls the tenant is passed in the URL and searchParameters are supplied as the request body.

See also

SearchParameters

Parameters:

request (MetadataSearchRequest) –

Return type:

MetadataSearchResponse

updateObject(request)

Update an existing object in the TRAC metadata store.

To save a new version, supply a metadata write request with the tenant code, object type and a tag for the item you want to save. The tag should contain all the attributes you want to set and the updated object definition, which must be supplied with the header from the previous version of the object. TRAC will apply the new version number when it creates the new version of the object. Only the latest version of an object can be updated and only CUSTOM objects can be updated by this API call.

Validation is performed on new objects before they are saved. Objects can fail validation because they are semantically invalid (e.g. missing required fields or inconsistencies within the object definition) or due to consistency errors (referring to other metadata items that don’t exist or don’t meet certain requirements). For new versions, validation also covers checking for compatibility with the previous version of the object.

NOTE: Validation is only partially implemented in the current release.

The call returns an ID response with the ID object and the newly assigned version number, as well as the tag version (which will always be 1).

Error conditions include: Invalid request, unknown tenant, unknown object ID or version, wrong object type (not the same as the prior version), object version is superseded.

Parameters:

request (MetadataWriteRequest) –

Return type:

tracdap.metadata.TagHeader

updateObjectBatch(request)

Update existing objects in batch in the TRAC metadata store.

To save a new version, supply a metadata write request with the tenant code, object type and a tag for the item you want to save. The tag should contain all the attributes you want to set and the updated object definition, which must be supplied with the header from the previous version of the object. TRAC will apply the new version number when it creates the new version of the object. Only the latest version of an object can be updated and only CUSTOM objects can be updated by this API call.

Validation is performed on new objects before they are saved. Objects can fail validation because they are semantically invalid (e.g. missing required fields or inconsistencies within the object definition) or due to consistency errors (referring to other metadata items that don’t exist or don’t meet certain requirements). For new versions, validation also covers checking for compatibility with the previous version of the object.

NOTE: Validation is only partially implemented in the current release.

The call returns an ID response with the ID object and the newly assigned version number, as well as the tag version (which will always be 1).

Error conditions include: Invalid request, unknown tenant, unknown object ID or version, wrong object type (not the same as the prior version), object version is superseded.

Parameters:

request (MetadataWriteBatchRequest) –

Return type:

MetadataWriteBatchResponse

updateTag(request)

Update the tag for an existing object in the TRAC metadata store.

To save a new tag, supply a metadata write request with the tenant code, object type and a tag for the item you want to save. The tag should contain all the attributes you want to include in the new tag, even if they have not been changed. The object definition must be supplied with a header only, supplying an object body is an invalid request. Tags for any type of object can be updated by this API call. Only the latest version of a tag can be updated.

The call returns an ID response with the ID and version of the object and the newly assigned tag number.

Error conditions include: Invalid request, unknown tenant, unknown object ID, object version or tag version, wrong object type (does not match what is stored in TRAC), tag version is superseded.

Parameters:

request (MetadataWriteRequest) –

Return type:

tracdap.metadata.TagHeader

updateTagBatch(request)

Update tags in batch for existing objects in the TRAC metadata store.

To save a new tag, supply a metadata write request with the tenant code, object type and a tag for the item you want to save. The tag should contain all the attributes you want to include in the new tag, even if they have not been changed. The object definition must be supplied with a header only, supplying an object body is an invalid request. Tags for any type of object can be updated by this API call. Only the latest version of a tag can be updated.

The call returns an ID response with the ID and version of the object and the newly assigned tag number.

Error conditions include: Invalid request, unknown tenant, unknown object ID, object version or tag version, wrong object type (does not match what is stored in TRAC), tag version is superseded.

Parameters:

request (MetadataWriteBatchRequest) –

Return type:

MetadataWriteBatchResponse

class tracdap.api.TracOrchestratorApi
cancelJob(request)
Parameters:

request (JobStatusRequest) –

Return type:

JobStatus

checkJob(request)
Parameters:

request (JobStatusRequest) –

Return type:

JobStatus

followJob(request)
Parameters:

request (JobStatusRequest) –

Return type:

JobStatus

submitJob(request)
Parameters:

request (JobRequest) –

Return type:

JobStatus

validateJob(request)
Parameters:

request (JobRequest) –

Return type:

JobStatus