10. Runtime Metadata

This tutorial is based on example code which can be found in the TRAC GitHub Repository under examples/models/python.

Models can look up the TRAC D.A.P. metadata associated with their inputs while they run, rather than relying only on the data itself. This example builds a Markdown report describing each input dataset, using both the runtime metadata and the schema.

Accessing metadata at runtime

get_metadata() returns a RuntimeMetadata object for a named input, giving access to its TRAC object ID and any metadata attributes that were recorded when the dataset was created.

src/tutorial/runtime_metadata.py
45    def run_model(self, ctx: trac.TracContext):
46
47        customer_loans = ctx.get_pandas_table("customer_loans")
48        customer_loans_schema = ctx.get_schema("customer_loans")
49        customer_loans_metadata = ctx.get_metadata("customer_loans")
50
51        account_filter = ctx.get_pandas_table("account_filter")
52        account_filter_schema = ctx.get_schema("account_filter")
53        account_filter_metadata = ctx.get_metadata("account_filter")
54
55        with ctx.put_file_stream("data_report") as report_bytes:
56            with io.TextIOWrapper(report_bytes, encoding='utf-8') as report:
57
58                report.write(f"# Data Report\n\n")
59
60                self.write_dataset_report("customer_loans", customer_loans_metadata, customer_loans_schema, customer_loans, report)
61                self.write_dataset_report("account_filter", account_filter_metadata, account_filter_schema, account_filter, report)

Note

Metadata is only available for inputs that come from real TRAC objects. Calling get_metadata() for a model parameter, an output before the job completes, or an input supplied as an intermediate result from another model in a flow, will return None rather than raising an error.

Building a report

Combining get_metadata() with get_schema() gives enough information to describe a dataset without needing to know its structure in advance - the object ID and attributes come from the metadata, while the field names and types come from the schema. The report itself is written out using put_file_stream(), the same file output method used in Using Files.

63    @classmethod
64    def write_dataset_report(
65            cls, dataset_name: str, metadata: trac.RuntimeMetadata,
66            schema: trac.SchemaDefinition, dataset: pd.DataFrame,
67            report: tp.TextIO):
68
69        report.write(f"## {dataset_name}\n\n")
70
71        if metadata is not None:
72
73            report.write(f"Object ID: {metadata.objectId.objectId}  \n")
74            report.write(f"Object version: {metadata.objectId.objectVersion}  \n\n")
75
76            report.write("Attributes:\n\n")
77
78            report.write("| Key | Value |\n")
79            report.write("| --- | ----- |\n")
80
81            for attr_key, attr_value in metadata.attributes.items():
82                report.write(f"| {attr_key} | {str(attr_value)} |\n")
83
84            report.write("\n")
85
86        else:
87            report.write("Metadata not available\n\n")
88
89        report.write("Columns:\n\n")
90
91        report.write("| Column | TRAC Type | Categorical | Pandas DType |\n")
92        report.write("| ------ | --------- | ----------- | ------------ |\n")
93
94        for field in schema.table.fields:
95            report.write(f"| {field.fieldName} | {field.fieldType.name} | {field.categorical} | {dataset[field.fieldName].dtype} |\n")
96
97        report.write(f"\nRows: {len(dataset)  }\n\n")

This pattern - reading metadata and schema alongside the data itself - is useful any time a model needs to behave generically across a range of datasets, such as building data quality reports or generic monitoring tools, without hard-coding assumptions about what any particular input contains.

See also

Full source code is available for the Runtime Metadata example on GitHub.