tracdap.rt.metadata

Submodules

Package Contents

Classes

ArrayValue

An array value holds an array of other Values.

BasicType

Basic types provide the set of core types available in the TRAC type system.

CopyStatus

Create a collection of name/value pairs.

CustomDefinition

DataDefinition

DateValue

Represent a date value.

DatetimeValue

Represent a date-time value.

DecimalValue

Represent a decimal value.

FieldSchema

FileDefinition

FlowDefinition

FlowEdge

FlowNode

FlowNodeType

Create a collection of name/value pairs.

FlowSocket

ImportModelJob

IncarnationStatus

Create a collection of name/value pairs.

JobDefinition

JobStatusCode

Create a collection of name/value pairs.

JobType

Create a collection of name/value pairs.

LogicalExpression

Logical expression for a search of the TRAC metadata store.

LogicalOperator

Metadata search logical operator, used as part of a LogicalExpression.

MapValue

A map value holds a map of string keys to other Values.

MetadataFormat

Available formats for representing the TRAC metadata.

MetadataVersion

Explicit versioning of the metadata schema.

ModelDefinition

Define a model for execution on the TRAC platform

ModelInputSchema

Describes the data schema of a model input

ModelOutputSchema

Describes the data schema of a model output

ModelParameter

Describes an individual parameter of a model

ObjectDefinition

Object definitions are the core structural element of TRAC's metadata model

ObjectType

Enumeration of TRAC's core object types.

PartKey

PartType

Create a collection of name/value pairs.

RunFlowJob

RunModelJob

SchemaDefinition

A schema definition describes the schema of a dataset

SchemaType

Enumeration of the available types of data schema

SearchExpression

Search expression for a search of the TRAC metadata store.

SearchOperator

Metadata search term operator, used as part of a SearchTerm

SearchParameters

Parameters to define a metadata search.

SearchTerm

Individual search term for a search of the TRAC metadata store.

StorageCopy

StorageDefinition

StorageIncarnation

StorageItem

TableSchema

Tag

Tags are the core informational element of TRAC's metadata model.

TagHeader

A tag header describes the identity and version of an object.

TagOperation

Enumeration of available TagUpdate operations.

TagSelector

A tag selector describes the selection of a unique object at a point in time.

TagUpdate

A tag update is a request for a single update operation on a tag.

TenantInfo

Information about a tenant that is set up on the TRAC platform.

TypeDescriptor

A type descriptor describes a data type used in the TRAC platform.

Value

A value expressed in the TRAC type system.

class tracdap.rt.metadata.ArrayValue

An array value holds an array of other Values.

All items in an array must have the same type.

See also

ARRAY

items

items

Type:

List[Value]

class tracdap.rt.metadata.BasicType(*args, **kwds)

Bases: enum.Enum

Basic types provide the set of core types available in the TRAC type system.

ARRAY = (8, 'An array of values, which may be primitive or composite values.\n    \n    All items in an...
BASIC_TYPE_NOT_SET = (0,)
BOOLEAN = (1, 'A true/false value')
DATE = (6, 'A date value.\n    \n    Dates do not take any account of time zones or offsets from UTC.\n    ')
DATETIME = (7, 'A date-time value.\n    \n    Date-time values may be expressed with an offset from UTC, as...
DECIMAL = (5, 'A fixed-point decimal value with known precision and scale.\n    \n    The available...
FLOAT = (3, "64 bit signed floating point number (referred to as 'double' in many languages)")
INTEGER = (2, '64 bit signed integer')
MAP = (9, 'An key-value map with string keys, values may be primitive or composite values.\n    \n   ...
STRING = (4, 'UTF encoded string value of arbitrary length.\n    \n    The encoding used (e.g. UTF-8,...
class tracdap.rt.metadata.CopyStatus(*args, **kwds)

Bases: enum.Enum

Create a collection of name/value pairs.

Example enumeration:

>>> class Color(Enum):
...     RED = 1
...     BLUE = 2
...     GREEN = 3

Access them by:

  • attribute access:

>>> Color.RED
<Color.RED: 1>
  • value lookup:

>>> Color(1)
<Color.RED: 1>
  • name lookup:

>>> Color['RED']
<Color.RED: 1>

Enumerations can be iterated over, and know how many members they have:

>>> len(Color)
3
>>> list(Color)
[<Color.RED: 1>, <Color.BLUE: 2>, <Color.GREEN: 3>]

Methods can be added to enumerations, and members can have their own attributes – see the documentation for details.

COPY_AVAILABLE = (1,)
COPY_EXPUNGED = (2,)
COPY_STATUS_NOT_SET = (0,)
class tracdap.rt.metadata.CustomDefinition
customData

customData

Type:

bytes

customSchemaType

customSchemaType

Type:

str

customSchemaVersion

customSchemaVersion

Type:

int

class tracdap.rt.metadata.DataDefinition
class Delta
dataItem

dataItem

Type:

str

deltaIndex

deltaIndex

Type:

int

class Part
partKey

partKey

Type:

PartKey

snap

snap

Type:

DataDefinition.Snap

class Snap
deltas

deltas

Type:

List[DataDefinition.Delta]

snapIndex

snapIndex

Type:

int

parts

parts

Type:

Dict[str, DataDefinition.Part]

schema

schema

Type:

Optional[SchemaDefinition]

schemaId

schemaId

Type:

Optional[TagSelector]

storageId

storageId

Type:

TagSelector

class tracdap.rt.metadata.DateValue

Represent a date value.

Dates are represented as strings in ISO 8601 format.

See also

DATE

isoDate

isoDate

Type:

str

class tracdap.rt.metadata.DatetimeValue

Represent a date-time value.

Date-times are represented as strings in ISO 8601 format.

See also

DATETIME

isoDatetime

isoDatetime

Type:

str

class tracdap.rt.metadata.DecimalValue

Represent a decimal value.

See also

DECIMAL

decimal

decimal

Type:

str

class tracdap.rt.metadata.FieldSchema
businessKey

businessKey

Type:

bool

categorical

categorical

Type:

bool

fieldName

fieldName

Type:

str

fieldOrder

fieldOrder

Type:

int

fieldType

fieldType

Type:

BasicType

formatCode

formatCode

Type:

Optional[str]

label

label

Type:

str

notNull

This could become mandatory with the next metadata update

Type:

Optional[bool]

class tracdap.rt.metadata.FileDefinition
dataItem

dataItem

Type:

str

extension

extension

Type:

str

mimeType

mimeType

Type:

str

name

name

Type:

str

size

size

Type:

int

storageId

storageId

Type:

TagSelector

class tracdap.rt.metadata.FlowDefinition
edges

edges

Type:

List[FlowEdge]

inputs

inputs

Type:

Dict[str, ModelInputSchema]

nodes

nodes

Type:

Dict[str, FlowNode]

outputs

outputs

Type:

Dict[str, ModelOutputSchema]

parameters

parameters

Type:

Dict[str, ModelParameter]

class tracdap.rt.metadata.FlowEdge
source

source

Type:

FlowSocket

target

target

Type:

FlowSocket

class tracdap.rt.metadata.FlowNode
inputs

inputs

Type:

List[str]

label

label

Type:

str

nodeAttrs

nodeAttrs

Type:

List[TagUpdate]

nodeSearch

nodeSearch

Type:

SearchExpression

nodeType

nodeType

Type:

FlowNodeType

outputs

outputs

Type:

List[str]

class tracdap.rt.metadata.FlowNodeType(*args, **kwds)

Bases: enum.Enum

Create a collection of name/value pairs.

Example enumeration:

>>> class Color(Enum):
...     RED = 1
...     BLUE = 2
...     GREEN = 3

Access them by:

  • attribute access:

>>> Color.RED
<Color.RED: 1>
  • value lookup:

>>> Color(1)
<Color.RED: 1>
  • name lookup:

>>> Color['RED']
<Color.RED: 1>

Enumerations can be iterated over, and know how many members they have:

>>> len(Color)
3
>>> list(Color)
[<Color.RED: 1>, <Color.BLUE: 2>, <Color.GREEN: 3>]

Methods can be added to enumerations, and members can have their own attributes – see the documentation for details.

INPUT_NODE = (1,)
MODEL_NODE = (3,)
NODE_TYPE_NOT_SET = (0,)
OUTPUT_NODE = (2,)
class tracdap.rt.metadata.FlowSocket
node

node

Type:

str

socket

socket

Type:

str

class tracdap.rt.metadata.ImportModelJob
entryPoint

entryPoint

Type:

str

language

language

Type:

str

modelAttrs

modelAttrs

Type:

List[TagUpdate]

package

package

Type:

str

packageGroup

packageGroup

Type:

Optional[str]

path

path

Type:

str

repository

repository

Type:

str

version

version

Type:

str

class tracdap.rt.metadata.IncarnationStatus(*args, **kwds)

Bases: enum.Enum

Create a collection of name/value pairs.

Example enumeration:

>>> class Color(Enum):
...     RED = 1
...     BLUE = 2
...     GREEN = 3

Access them by:

  • attribute access:

>>> Color.RED
<Color.RED: 1>
  • value lookup:

>>> Color(1)
<Color.RED: 1>
  • name lookup:

>>> Color['RED']
<Color.RED: 1>

Enumerations can be iterated over, and know how many members they have:

>>> len(Color)
3
>>> list(Color)
[<Color.RED: 1>, <Color.BLUE: 2>, <Color.GREEN: 3>]

Methods can be added to enumerations, and members can have their own attributes – see the documentation for details.

INCARNATION_AVAILABLE = (1,)
INCARNATION_EXPUNGED = (2,)
INCARNATION_STATUS_NOT_SET = (0,)
class tracdap.rt.metadata.JobDefinition
importModel

importModel

Type:

Optional[ImportModelJob]

jobType

jobType

Type:

JobType

runFlow

runFlow

Type:

Optional[RunFlowJob]

runModel

runModel

Type:

Optional[RunModelJob]

class tracdap.rt.metadata.JobStatusCode(*args, **kwds)

Bases: enum.Enum

Create a collection of name/value pairs.

Example enumeration:

>>> class Color(Enum):
...     RED = 1
...     BLUE = 2
...     GREEN = 3

Access them by:

  • attribute access:

>>> Color.RED
<Color.RED: 1>
  • value lookup:

>>> Color(1)
<Color.RED: 1>
  • name lookup:

>>> Color['RED']
<Color.RED: 1>

Enumerations can be iterated over, and know how many members they have:

>>> len(Color)
3
>>> list(Color)
[<Color.RED: 1>, <Color.BLUE: 2>, <Color.GREEN: 3>]

Methods can be added to enumerations, and members can have their own attributes – see the documentation for details.

CANCELLED = (10,)
FAILED = (9,)
FINISHING = (7,)
JOB_STATUS_CODE_NOT_SET = (0,)
PENDING = (3,)
PREPARING = (1,)
QUEUED = (4,)
RUNNING = (6,)
SUBMITTED = (5,)
SUCCEEDED = (8,)
VALIDATED = (2,)
class tracdap.rt.metadata.JobType(*args, **kwds)

Bases: enum.Enum

Create a collection of name/value pairs.

Example enumeration:

>>> class Color(Enum):
...     RED = 1
...     BLUE = 2
...     GREEN = 3

Access them by:

  • attribute access:

>>> Color.RED
<Color.RED: 1>
  • value lookup:

>>> Color(1)
<Color.RED: 1>
  • name lookup:

>>> Color['RED']
<Color.RED: 1>

Enumerations can be iterated over, and know how many members they have:

>>> len(Color)
3
>>> list(Color)
[<Color.RED: 1>, <Color.BLUE: 2>, <Color.GREEN: 3>]

Methods can be added to enumerations, and members can have their own attributes – see the documentation for details.

IMPORT_DATA = (4,)
IMPORT_MODEL = (3,)
JOB_TYPE_NOT_SET = (0,)
RUN_FLOW = (2,)
RUN_MODEL = (1,)
class tracdap.rt.metadata.LogicalExpression

Logical expression for a search of the TRAC metadata store.

Applies a logical operator to one or more sub-expressions.

expr

A set of sub-expressions.

For AND or OR operations, there must be two or more sub-expressions. For NOT operations, there must be precisely one sub-expression.

Type:

List[SearchExpression]

operator

The logical operator to apply to sub-expressions

Type:

LogicalOperator

class tracdap.rt.metadata.LogicalOperator(*args, **kwds)

Bases: enum.Enum

Metadata search logical operator, used as part of a LogicalExpression.

AND = (1, 'LOGICAL AND\n    \n    The AND operator combines two or more search expressions, the...
LOGICAL_OPERATOR_NOT_SET = (0,)
NOT = (3, 'LOGICAL NOT\n    \n    The NOT operator applies to a single sub-expression, the logical...
OR = (2, 'LOGICAL OR\n    \n    The OR operator combines two or more search expressions, the logical...
class tracdap.rt.metadata.MapValue

A map value holds a map of string keys to other Values.

Maps may be uniform (holding all the same value type) or non-uniform (holding mixed value types) depending on the type descriptor of the Value that contains them.

See also

MAP, TypeDescriptor

entries

entries

Type:

Dict[str, Value]

class tracdap.rt.metadata.MetadataFormat(*args, **kwds)

Bases: enum.Enum

Available formats for representing the TRAC metadata.

Use for communication between components, config files metadata stored in the metadata database.

JSON = (2,)
METADATA_FORMAT_NOT_SET = (0,)
PROTO = (1,)
YAML = (3,)
class tracdap.rt.metadata.MetadataVersion(*args, **kwds)

Bases: enum.Enum

Explicit versioning of the metadata schema.

A strictly increasing enumeration of metadata versions. The special value CURRENT is always set to the latest version and used by default, in API calls, config files and for storing in the metadata database.

TRAC can use this information identify and handle old metadata found in the metadata database. In future it may also be possible to request old metadata versions in API calls, or to run upgrades of metadata stored in an older metadata format.

CURRENT = (1,)
METADATA_VERSION_NOT_SET = (0,)
V1 = (1,)
class tracdap.rt.metadata.ModelDefinition

Define a model for execution on the TRAC platform

entryPoint

entryPoint

Type:

str

inputs

inputs

Type:

Dict[str, ModelInputSchema]

language

language

Type:

str

outputs

outputs

Type:

Dict[str, ModelOutputSchema]

package

package

Type:

str

packageGroup

packageGroup

Type:

Optional[str]

parameters

parameters

Type:

Dict[str, ModelParameter]

path

path

Type:

Optional[str]

repository

repository

Type:

str

staticAttributes

Static attributes defined in model code

Type:

Dict[str, Value]

version

version

Type:

str

class tracdap.rt.metadata.ModelInputSchema

Describes the data schema of a model input

In many cases models define the entire schemas of their inputs, in which case the input schema is just a wrapper around a schema definition. This is what is supported now.

Other approaches are possible. Models can define dynamic inputs, in which case the input schema is provided at runtime and can be interrogated by the model code. Models may also define inputs with some required fields and a dynamic portion. For non-tabular inputs, other options may be required. These capabilities may be added in future releases.

label

label

Type:

Optional[str]

schema

schema

Type:

SchemaDefinition

class tracdap.rt.metadata.ModelOutputSchema

Describes the data schema of a model output

In many cases models define the entire schemas of their outputs, in which case the output schema is just a wrapper around a schema definition. This is what is supported now.

Other approaches are possible. Models can define dynamic outputs, in which case the model decides at runtime what the output schema will be. Pass-through schemas (output X has the same schema as dynamic input Y) and pass-through-extend schemas (output X has the schema of dynamic input Y, plus one or more new columns) can also be useful. These capabilities may be added in future releases.

label

label

Type:

Optional[str]

schema

schema

Type:

SchemaDefinition

class tracdap.rt.metadata.ModelParameter

Describes an individual parameter of a model

defaultValue

defaultValue

Type:

Optional[Value]

label

label

Type:

str

paramType

paramType

Type:

TypeDescriptor

class tracdap.rt.metadata.ObjectDefinition

Object definitions are the core structural element of TRAC’s metadata model

Definitions describe every object that is stored in the TRAC platform and there is a one-to-one relation between definitions and objects. I.e. every dataset has its own data definition, every model has its own model definition and so on. Definitions also describe actions that take place on the platform by way of job definitions, so a “job” is just another type of object. Each type of object has its own definition and definitions can be added or extended as the platform evolves.

The object definition container class allows different types of objects to be stored, indexed and accessed in the same way. Every object has a standard object header which contains enough information to identify the object.

TRAC object definitions can be versioned. In order to use versioning the semantics of versioning must be defined and those vary depending on the object type. Currently these semantics are defined for DATA objects, see DataDefinition for details. Versioning is also allowed for CUSTOM objects, in this case it is the responsibility of the application to define versioning semantics. Versioning is not currently permitted for other object types.

Object definitions are intended for storing structural data necessary to access data and run jobs on the TRAC platform. Informational data to catalogue and describe objects is stored in tags. Tags are a lot more flexible than object definitions, so applications built on the TRAC platform may choose to store structural information in tags where their required structure is not supported by TRAC’s core object definitions.

See also

Tag

custom

custom

Type:

Optional[CustomDefinition]

data

data

Type:

Optional[DataDefinition]

file

file

Type:

Optional[FileDefinition]

flow

flow

Type:

Optional[FlowDefinition]

job

job

Type:

Optional[JobDefinition]

model

model

Type:

Optional[ModelDefinition]

objectType

objectType

Type:

ObjectType

schema

schema

Type:

Optional[SchemaDefinition]

storage

storage

Type:

Optional[StorageDefinition]

class tracdap.rt.metadata.ObjectType(*args, **kwds)

Bases: enum.Enum

Enumeration of TRAC’s core object types.

See also

ObjectDefinition

CUSTOM = (6,)
DATA = (1,)
FILE = (5,)
FLOW = (3,)
JOB = (4,)
MODEL = (2,)
OBJECT_TYPE_NOT_SET = (0,)
SCHEMA = (8,)
STORAGE = (7,)
class tracdap.rt.metadata.PartKey
opaqueKey

opaqueKey

Type:

str

partRangeMax

partRangeMax

Type:

Optional[Value]

partRangeMin

partRangeMin

Type:

Optional[Value]

partType

partType

Type:

PartType

partValues

partValues

Type:

List[Value]

class tracdap.rt.metadata.PartType(*args, **kwds)

Bases: enum.Enum

Create a collection of name/value pairs.

Example enumeration:

>>> class Color(Enum):
...     RED = 1
...     BLUE = 2
...     GREEN = 3

Access them by:

  • attribute access:

>>> Color.RED
<Color.RED: 1>
  • value lookup:

>>> Color(1)
<Color.RED: 1>
  • name lookup:

>>> Color['RED']
<Color.RED: 1>

Enumerations can be iterated over, and know how many members they have:

>>> len(Color)
3
>>> list(Color)
[<Color.RED: 1>, <Color.BLUE: 2>, <Color.GREEN: 3>]

Methods can be added to enumerations, and members can have their own attributes – see the documentation for details.

PART_BY_RANGE = (1,)
PART_BY_VALUE = (2,)
PART_ROOT = (0,)
class tracdap.rt.metadata.RunFlowJob
flow

flow

Type:

TagSelector

inputs

inputs

Type:

Dict[str, TagSelector]

models

models

Type:

Dict[str, TagSelector]

outputAttrs

outputAttrs

Type:

List[TagUpdate]

outputs

outputs

Type:

Dict[str, TagSelector]

parameters

parameters

Type:

Dict[str, Value]

priorOutputs

priorOutputs

Type:

Dict[str, TagSelector]

class tracdap.rt.metadata.RunModelJob
inputs

inputs

Type:

Dict[str, TagSelector]

model

model

Type:

TagSelector

outputAttrs

outputAttrs

Type:

List[TagUpdate]

outputs

outputs

Type:

Dict[str, TagSelector]

parameters

parameters

Type:

Dict[str, Value]

priorOutputs

priorOutputs

Type:

Dict[str, TagSelector]

class tracdap.rt.metadata.SchemaDefinition

A schema definition describes the schema of a dataset

Schema definitions can be top level objects (a type of object definition), in which case they can be referred to by multiple data definitions. Alternatively they can be embedded in a data definition to create datasets with one-off schemas.

A table schema describes the schema of a tabular data set. Other schema types may be added later, e.g. for matrices, tensors, curves, surfaces and structured datasets.

See also

DataDefinition

partType

partType

Type:

PartType

schemaType

schemaType

Type:

SchemaType

table

table

Type:

Optional[TableSchema]

class tracdap.rt.metadata.SchemaType(*args, **kwds)

Bases: enum.Enum

Enumeration of the available types of data schema

Currently only table schemas are supported, other schema types may be added later.

See also

SchemaDefinition

SCHEMA_TYPE_NOT_SET = (0,)
TABLE = (1,)
class tracdap.rt.metadata.SearchExpression

Search expression for a search of the TRAC metadata store.

A search expression is either a single search term or a logical combination of other expressions. Search expressions can be built up to express complex logical conditions.

logical

Set if this search expression is a logical expression

Type:

Optional[LogicalExpression]

term

Set if this search expression is a single term

Type:

Optional[SearchTerm]

class tracdap.rt.metadata.SearchOperator(*args, **kwds)

Bases: enum.Enum

Metadata search term operator, used as part of a SearchTerm

See also

SearchTerm

EQ = (1, 'EQUALS\n    \n    The EQ operator matches a tag when the tag has an attribute that matches...
EXISTS = (8, 'EXISTS\n    \n    If an attribute type is provided the EXISTS operator matches a tag of...
GE = (6, 'GREATER THAN OR EQUAL TO\n    \n    The GE operator matches a tag when the tag has an...
GT = (5, 'GREATER THAN\n    \n    The GT operator matches a tag when the tag has an attribute with a...
IN = (7, 'IN\n    \n    The IN operator matches a tag when the tag has an attribute whose value is...
LE = (4, 'LESS THAN OR EQUAL TO\n    \n    The LE operator matches a tag when the tag has an...
LT = (3, 'LESS THAN\n    \n    The LT operator matches a tag when the tag has an attribute with a...
NE = (2, 'DOES NOT EQUAL\n    \n    The NE operator matches a tag precisely when the EQ operator does...
SEARCH_OPERATOR_NOT_SET = (0,)
class tracdap.rt.metadata.SearchParameters

Parameters to define a metadata search.

objectType

The type of object to search for

Type:

ObjectType

priorTags

Include prior tags in the search.

By default, only the latest tag for each object version is considered in a search. If the as-of parameter is specified, latest tags are considered as of that time.

Setting this flag to true will cause TRAC to consider superseded tags in the search. If the as-of parameter is specified as well then all tags up to that time are considered. Only the latest matching tag will be included in the search result.

This flag can be combined with priorVersions to search across all tags and object versions. If neither flag is specified, only the latest version and latest tag is considered for each object.

Type:

bool

priorVersions

Include prior versions of objects in the search.

By default, only the latest version of each object is considered in a search. If the as-of parameter is specified, latest versions are considered as of that time.

Setting this flag to true will cause TRAC to consider superseded object versions in the search. If the as-of parameter is specified as well then all object versions up to that time are considered. Only the latest matching version will be included in the search result.

This flag can be combined with priorTags to search across all tags and object versions. If neither flag is specified, only the latest version and latest tag is considered for each object.

Type:

bool

search

A search expression based on tag attributes.

This field is optional. If no search parameters are given, then all objects are returned.

Type:

SearchExpression

searchAsOf

Perform the search as of a specific date/time.

Supplying this field will cause TRAC to ignore all metadata changes from the specified time onwards. The result will be the same as if a search was performed at the specified time with this field left blank.

If a zone offset is supplied as part of the timestamp, TRAC will apply the offset to search across all metadata items in UTC.

If this parameter is not supplied, the search will be executed as of the current time.

Type:

Optional[DatetimeValue]

class tracdap.rt.metadata.SearchTerm

Individual search term for a search of the TRAC metadata store.

Applies a search operator against an individual tag attribute.

attrName

The name of the attribute to search for

Type:

str

attrType

The primitive type of the attribute being searched for

Type:

BasicType

operator

The search operator to apply

Type:

SearchOperator

searchValue

The search value to look for

Type:

Value

class tracdap.rt.metadata.StorageCopy
copyStatus

copyStatus

Type:

CopyStatus

copyTimestamp

copyTimestamp

Type:

DatetimeValue

storageFormat

storageFormat

Type:

str

storageKey

storageKey

Type:

str

storageOptions

storageOptions

Type:

Dict[str, Value]

storagePath

storagePath

Type:

str

class tracdap.rt.metadata.StorageDefinition
dataItems

dataItems

Type:

Dict[str, StorageItem]

storageOptions

storageOptions

Type:

Dict[str, Value]

class tracdap.rt.metadata.StorageIncarnation
copies

copies

Type:

List[StorageCopy]

incarnationIndex

incarnationIndex

Type:

int

incarnationStatus

incarnationStatus

Type:

IncarnationStatus

incarnationTimestamp

incarnationTimestamp

Type:

DatetimeValue

class tracdap.rt.metadata.StorageItem
incarnations

incarnations

Type:

List[StorageIncarnation]

class tracdap.rt.metadata.TableSchema
fields

fields

Type:

List[FieldSchema]

class tracdap.rt.metadata.Tag

Tags are the core informational element of TRAC’s metadata model.

A tag is a set of attributes (key-value pairs) associated with an object definition, intended for storing descriptive and informational data as well as application-level metadata that is not part of the object definition model. Here is an example of a set of tag attributes to illustrate some ways they can be used:

# A descriptive field intended for human users.

display_name: "Customer accounts for March 2020, corrected April 6th"

# A classification that can be used for searching or indexing.
# Client applications can also use this to find datasets of a certain
# type; typically an application will define a set of attributes that are
# "structural", i.e. the application uses those attributes to decide which
# objects to present for certain purposes.

dataset_class: "customer_accounts"

# Properties of an item can be added as individual attributes so they can
# be searched and displayed individually. This avoids the anti-pattern of
# putting multiple attributes into a single name/label field:
#    customer_accounts_mar20_scotland_commercial_approved

accounting_date: (DATE) 2020-03-31
region: "Scotland"
book: "commercial_property"
figures_approved: (BOOLEAN) true

# Attributes can be multi-valued. This can be helpful for applying
# regulatory classifiers, where multiple classifiers may apply to a
# single item.

data_classification: ["confidential", "gdpr_pii", "audited"]

# TRAC records a number of "controlled" attributes, these are set by the
# platform and cannot be modified directly through the metadata API.
# Controlled attributes start with the prefix "trac_".

trac_create_time: (DATETIME) 2020-04-01 10:37:05
trac_create_user_id: "jane.doe"
trac_create_user_name: "Jane Doe"

Tags use immutable versioning in the same way as objects - each version of a tag is immutable and “updating” a tag means creating a new version with one or more modified attributes. Each version of an object has its own series of tags starting at tag version 1.

As an example of this versioning, consider a partitioned dataset with daily account records. Version X of the dataset contains data up to a certain date and might have a tag saying it is signed off. A user/process then adds a new partition with the next day’s data, creating version X+1. In this case, object version X would still be signed off while version X+1 is awaiting approval. When version X+1 is approved, the tag for that version can be “updated”. The application could decide whether to show the most recent version of the data, or an earlier version that has the sign-off attribute set.

attrs

Tag attributes are key-value pairs where the value is a metadata Value.

Attribute values are restricted to primitive types (which are interpreted as single-valued attributes) or arrays of primitive types (which are interpreted as multi-valued attributes). Any attribute may be single- or multi-valued, except BOOLEAN attributes which are always single-valued.

An attribute may change from single- to multi-valued or vice versa when a tag is updated, e.g. is a classification is added or removed. An array containing a single item is treated as a single-valued attribute, i.e. there is no distinction between a single value and an array of one item. Single-valued attributes are always returned as primitive types when querying the metadata API.

Single- and multi-valued attributes have different search semantics. For example, inequalities are not defined on multi-valued attributes. See SearchParameters for more details.

See also

SearchParameters

Type:

Dict[str, Value]

definition

The object definition that the tag is associated with.

Sometimes the definition may be omitted, for example the results of a metadata search include only headers and attributes.

See also

ObjectDefinition

Type:

Optional[ObjectDefinition]

header

The tag header uniquely identifies the current tag and the object it is associated with.

See also

TagHeader

Type:

TagHeader

class tracdap.rt.metadata.TagHeader

A tag header describes the identity and version of an object.

See also

Tag, ObjectDefinition

isLatestObject

isLatest flag for the object the tag is associated with.

Type:

bool

isLatestTag

isLatest flag for the tag.

Type:

bool

objectId

Object ID of the object this tag is associated with.

Object IDs are UUIDs (RFC4122, https://www.ietf.org/rfc/rfc4122.txt)

Type:

str

objectTimestamp

Timestamp for when this version of the object was created.

Type:

DatetimeValue

objectType

Object type of the object this tag is associated with.

See also

ObjectType

Type:

ObjectType

objectVersion

Version of the object this tag is associated with.

Type:

int

tagTimestamp

Timestamp for when this version of the tag was created.

Type:

DatetimeValue

tagVersion

Version of this tag.

Type:

int

class tracdap.rt.metadata.TagOperation(*args, **kwds)

Bases: enum.Enum

Enumeration of available TagUpdate operations.

See also

TagUpdate

APPEND_ATTR = (4, 'Append one or more values to an existing attribute, fail if the attribute\n    does not...
CLEAR_ALL_ATTR = (6, 'Remove all the attributes from a tag.\n    \n    This operation does not affect controlled...
CREATE_ATTR = (2, 'Add an attribute to a tag, fail if the attribute already exists.')
CREATE_OR_APPEND_ATTR = (1, 'Add an attribute to a tag or append to it if it already exists.\n    \n    If the attribute...
CREATE_OR_REPLACE_ATTR = (0, 'Add an attribute to a tag or replace it if it already exists.\n    \n    This is the...
DELETE_ATTR = (5, 'Remove an attribute from a tag, fail if the attribute does not exist.')
REPLACE_ATTR = (3, 'Replace an attribute on a tag, fail if the attribute does not exist.\n    \n    When...
class tracdap.rt.metadata.TagSelector

A tag selector describes the selection of a unique object at a point in time.

A tag selector refers to a single object ID and provides criteria for selecting the object version and tag version. The available selection criteria are:

  • Select an explicit version number

  • Select the latest available version

  • Select the version that was live at specific point in time

A selector for an explicit version number will always match that exact version number. These “fixed” types of selector can be used to refer to elements of a repeatable job, because the versions they refer to will never change.

A selector for the latest version will select different versions over time, as they become available. These “variable” types of selector can be used by client applications that want to query the latest state of an object. If a job is set up using variable selectors, TRAC will convert them to fixed selectors for the particular versions that were selected before saving the job definition.

Criteria for object versions and tag versions can be “mixed and matched”, so e.g. latestObject = true with tagVersion = 1 is allowed.

See also

Tag, TagHeader

latestObject

Select the latest version of the object (the version that is live now).

If this flag is specified, it must be set to true.

Type:

Optional[bool]

latestTag

Select the latest version of the tag (the version that is live now).

If this flag is specified, it must be set to true.

Type:

Optional[bool]

objectAsOf

Select the version of the object that was live as of a particular point in time. Represented using ISO 8601.

Type:

Optional[DatetimeValue]

objectId

Object ID of the tag being selected.

Object IDs are UUIDs (RFC4122, https://www.ietf.org/rfc/rfc4122.txt)

Type:

str

objectType

Object type of the tag being selected.

See also

ObjectType

Type:

ObjectType

objectVersion

Select an explicit version of the object.

Type:

Optional[int]

tagAsOf

Select the version of the tag that was live as of a particular point in time. Represented using ISO 8601.

Type:

Optional[DatetimeValue]

tagVersion

Select an explicit version of the tag.

Type:

Optional[int]

class tracdap.rt.metadata.TagUpdate

A tag update is a request for a single update operation on a tag.

Tag updates can be supplied to TRAC via an API call to request updates to a tag. They may also be included in TRAC policy objects or client application logic, to describe a set of operations that is performed in response to a particular action.

See also

MetadataWriteRequest

attrName

Name of the attribute this update refers to.

This field must be supplied for operations that refer to a single attribute, otherwise it should be left blank.

Type:

str

operation

The operation requested in this update

See also

TagOperation

Type:

TagOperation

value

Attribute value to use for this update.

This field must be supplied for operations that use a value, otherwise it should be omitted.

See also

Value

Type:

Optional[Value]

class tracdap.rt.metadata.TenantInfo

Information about a tenant that is set up on the TRAC platform.

description
  • A short description of the tenant, suitable for displaying to users in lists.

Type:

str

tenantCode
  • Unique code used to identify the tenant, required by most API calls.

Type:

str

class tracdap.rt.metadata.TypeDescriptor

A type descriptor describes a data type used in the TRAC platform.

For complex types, the descriptor holds a full type description. E.g. for array types, the type being held in the array is described. At a later point, precision fields may be introduced for decimals, or field types for structs.

arrayType

For array types only, describe the type contained in the array.

Type:

Optional[TypeDescriptor]

basicType

The basic type being described.

Type:

BasicType

mapType

For map types only, describe the type contained in the map.

To describe a uniform map the mapType descriptor must be set to a valid type descriptor, in this case all values in the map must match the type of the descriptor. If mapType is not set or is present but has basicType = BASIC_TYPE_NOT_SET then the map is non-uniform, values must be inspected individually to determine their type.

Type:

Optional[TypeDescriptor]

class tracdap.rt.metadata.Value

A value expressed in the TRAC type system.

A value can express a primitive value, or a composite value such as an array. Arbitrary nesting of composite types is permitted, although most functions will limit the set of acceptable types during validation.

Values include a type descriptor field. For primitive values the type descriptor is optional. For composite types the root value must include a full valid type descriptor, i.e. a descriptor that goes down to the leaf types. Sub-values in a composite type are free to omit the type descriptor, even if there are multiple levels of nesting, however any extra descriptors that are provided must also be full and valid.

TRAC will always provide values with the type descriptors “normalised”. This means that a root value will always have a type descriptor (even if it is a primitive type) and sub-values will never have a type descriptor. It is not necessary or preferred for application code to send values to TRAC with normalised type descriptors, TRAC will always perform normalisation.

See also

TypeDescriptor

arrayValue

An array of Values.

All items in an array must have the same type.

See also

ARRAY

Type:

Optional[ArrayValue]

booleanValue

A boolean value.

Represented natively in both protobuf and JSON.

Type:

Optional[bool]

dateValue

A date value.

See also

DATE, DateValue

Type:

Optional[DateValue]

datetimeValue

A date-time value.

Date-times are represented as strings in ISO 8601 format.

Date-time values support nanosecond precision, however in practice the available precision may be less than this. In particular, tag attributes are always stored at microsecond precision. Values passed into models or used in application code will be limited to the precision supported by date-time types in their own coding language.

Time zone offsets are supported as described in ISO 8601. Date-time values should always include a zone offset to guarantee behaviour. If a zone offset is not supplied TRAC will normalized incoming values by applying a zone offset, currently the applied offset will be UTC.

Type:

Optional[DatetimeValue]

decimalValue

A decimal value with fixed precision and scale.

See also

DECIMAL

Type:

Optional[DecimalValue]

floatValue

A 64-bit signed floating point value.

Represented natively in both protobuf and JSON.

Type:

Optional[float]

integerValue

A 64-bit signed integer value.

Represented natively in both protobuf and JSON.

In JavaScript and JSON, native integers may be limited to less than full range of a 64 bit integer. This is because JavaScript has a single Number type that represents both integers and floats. The safe limit for integers that can be expressed without a risk of rounding in JavaScript is Number.MAX_SAFE_INTEGER.

Type:

Optional[int]

mapValue

A map of string keys to Values.

Maps may be uniform (holding all the same value type) or non-uniform (holding mixed value types) depending on the type descriptor of this Value.

See also

MAP, TypeDescriptor

Type:

Optional[MapValue]

stringValue

A string value.

Protobuf encodes strings as UTF-8 on the wire. String values in JSON requests are also be encoded as UTF-8 on the wire, as per RFC 8259. When reading or writing string values in application code, they will be presented in the normal encoding scheme of application’s coding language.

Type:

Optional[str]

type

Type descriptor for the current value.

A type descriptor is always required for values that are the root of a composite type, otherwise it is optional.

Type:

TypeDescriptor