5.3.3. xgt.TableFrame

class xgt.TableFrame(conn, name, schema, container_id, commit_id)

TableFrame object represent a table held on the xGT server. It can be used to retrieve information about it and should not be instantiated directly by the user. Methods that return this object: Connection.get_frame(), Connection.get_frames() and Connection.create_table_frame(). A table may also be created by a MATCH query and may contain query results.

Parameters:

  • conn (Connection) – An open connection to an xGT server.

  • name (str) – Fully qualified name of the frame, including the namespace.

  • schema (list of lists) – List of lists associating property names with xGT data types.

Examples

>>> import xgt
>>> conn = xgt.Connection()
>>> ... run query and store results in Results
>>> t = conn.get_frame('Results')
>>> print(t.name)

Methods

__init__(conn, name, schema, container_id, ...)

Constructor for TableFrame.

get_data([offset, length, columns, format, ...])

Returns frame data starting at a given offset and spanning a given length.

get_data_arrow([offset, length, ...])

Returns an Apache Arrow Table containing frame data starting at a given offset and spanning a given length.

get_data_pandas([offset, length, ...])

Returns a pandas DataFrame containing frame data starting at a given offset and spanning a given length.

insert(data[, row_labels, ...])

Inserts data rows.

load(paths[, header_mode, record_history, ...])

Loads data from one or more files specified in the list of paths.

save(path[, offset, length, headers, ...])

Writes the rows from the frame to a file in the location indicated by the path parameter.

Attributes

connection

The connection used when constructing the frame.

name

Name of the frame.

num_rows

Gets the number of rows in the frame.

row_label_universe

Gets the universe of row security labels that can be attached to rows of this frame.

schema

Gets the property names and types of the frame.
__init__(conn, name, schema, container_id, commit_id)

Constructor for TableFrame. Called when TableFrame is created.

property connection

The connection used when constructing the frame.

Type:

Connection object

get_data(offset=0, length=None, columns=None, format='python', include_row_labels=False, row_label_column_header=None, duration_as_interval=False)

Returns frame data starting at a given offset and spanning a given length.

Parameters:

  • offset (int) – Position (index) of the first row to be retrieved. Optional. Default=0.

  • length (int) – Maximum number of rows to be retrieved starting from the row indicated by offset. A value of ‘None’ means ‘all rows’ on and after the offset. Optional. Default=None.

  • columns (list of int or str) –

    The columns to retrieve. Given as a mixed list of column positions and schema column names. A value of ‘None’ means all columns. Optional. Default=None.

    New in version 1.14.0.

  • format (str) –

    Selects the data format returned: a Python list of list, a pandas Dataframe, or an Apache Arrow Table. Must be one of ‘python’, ‘pandas’, or ‘arrow’. Optional. Default=’python’.

    New in version 1.14.0.

  • include_row_labels (bool) –

    Indicates whether the security labels for each row should be egested along with the row.

    New in version 1.5.0.

  • row_label_column_header (str) – The header column name to use for all row labels if include_row_labels is true and headers is true. Ignored for python format.

Return type:

list of lists, pandas DataFrame, or Apache Arrow Table

Raises:

  • XgtNameError – If the frame does not exist on the server.

  • XgtSecurityError – If the user does not have required permissions for this action.

  • XgtTransactionError – If a conflict with another transaction occurs.

  • ValueError – If parameter is out of bounds or invalid format given.

get_data_arrow(offset=0, length=None, include_row_labels=False, row_label_column_header=None, duration_as_interval=False)

Returns an Apache Arrow Table containing frame data starting at a given offset and spanning a given length.

New in version 1.11.0.

Deprecated since version 1.14.0: Use get_data instead.

Parameters:

  • offset (int) – Position (index) of the first row to be retrieved. Optional. Default=0.

  • length (int) – Maximum number of rows to be retrieved starting from the row indicated by offset. A value of ‘None’ means ‘all rows’ on and after the offset. Optional. Default=None.

  • include_row_labels (bool) – Indicates whether the security labels for each row should be egested along with the row.

  • row_label_column_header (str) – The header column name to use for all row labels if include_row_labels is true and headers is true.

Return type:

Apache Arrow Table

Raises:

  • XgtNameError – If the frame does not exist on the server.

  • XgtSecurityError – If the user does not have required permissions for this action.

  • XgtTransactionError – If a conflict with another transaction occurs.

  • ValueError – If parameter is out of bounds.

get_data_pandas(offset=0, length=None, include_row_labels=False, row_label_column_header=None)

Returns a pandas DataFrame containing frame data starting at a given offset and spanning a given length.

Deprecated since version 1.14.0: Use get_data instead.

Parameters:

  • offset (int) – Position (index) of the first row to be retrieved. Optional. Default=0.

  • length (int) – Maximum number of rows to be retrieved starting from the row indicated by offset. A value of ‘None’ means ‘all rows’ on and after the offset. Optional. Default=None.

  • include_row_labels (bool) –

    Indicates whether the security labels for each row should be egested along with the row.

    New in version 1.5.0.

  • row_label_column_header (str) –

    The header column name to use for all row labels if include_row_labels is true and headers is true.

    New in version 1.5.0.

Return type:

pandas DataFrame

Raises:

  • XgtNameError – If the frame does not exist on the server.

  • XgtSecurityError – If the user does not have required permissions for this action.

  • XgtTransactionError – If a conflict with another transaction occurs.

  • ValueError – If parameter is out of bounds.

insert(data, row_labels=None, row_label_columns=None, frame_to_input_column_mapping=None, suppress_errors=False, input_filter=None)

Inserts data rows. The properties of the new data must match the schema in both order and type.

Parameters:

  • data (list, pandas DataFrame, or pyarrow Table) – Data represented by a list of lists of data items, by a pandas DataFrame or by pyarrow Table.

  • row_labels (list) –

    A list of security labels to attach to each row inserted. Each label must have been passed in to the row_label_universe parameter when creating the frame. Note: Only one of row_labels and row_label_columns must be passed.

    New in version 1.5.0.

  • row_label_columns (list) –

    A list of integer column indices indicating which columns in the input data contain security labels to attach to the inserted row. Note: Only one of row_labels and row_label_columns must be passed.

    New in version 1.5.0.

  • frame_to_input_column_mapping (dictionary) –

    Maps the frame column names to input columns for the ingest. The key of each element is a frame column name. The value is either the name of the column (from the Pandas frame or xGT schema column name for lists) or the file column index.

    New in version 1.13.0.

  • suppress_errors (bool) –

    If true, will continue to insert data if an ingest error is encountered, placing the first 1000 errors in the job history. If false, stops on first error and raises. Defaults to False.

    New in version 1.11.0.

  • input_filter (str) –

    TQL fragment used to filter, modify and parameterize the raw data from the input to produce the row data fed to the frame.

    New in version 1.14.0.

Returns:

A Job object representing the job that has executed the insert.

Return type:

Job

Raises:

  • XgtIOError – If there are errors in the data being inserted or some data could not be inserted into the frame.

  • XgtNameError – If the frame does not exist on the server.

  • XgtSecurityError – If the user does not have required permissions for this action.

  • XgtTransactionError – If a conflict with another transaction occurs.

load(paths, header_mode='none', record_history=True, row_labels=None, row_label_columns=None, delimiter=',', frame_to_input_column_mapping=None, suppress_errors=False, frame_to_file_column_mapping=None, input_filter=None)

Loads data from one or more files specified in the list of paths. These files may be CSV, Parquet, or compressed CSV. Some limitations exist for compressed CSV. See docs.trovares.com for more details. Each path may have its own protocol as described below.

Parameters:

  • paths (list or str) –

    A single path or a list of paths to files. Local or server paths may contain wildcards. Wildcard expressions can contain *, ?, range sets, and negation. See docs.trovares.com for more details.

    Syntax for one file path

    Resource type

    Path syntax

    local to Python:

    ’<path to file>’ ‘xgt://<path to file>’

    xGT server:

    ’xgtd://<path to file>’

    AWS s3:

    ’s3://<path to file>’

    https site:

    ’https://<path to file>’

    http site:

    ’http://<path to file>’

    ftps server:

    ’ftps://<path to file>’

    ftp server:

    ’ftp://<path to file>’

  • header_mode (str) –

    Indicates how the file header should be processed:

    • HeaderMode.NONE: No header exists.

    • HeaderMode.IGNORE: Ignore the first line containing the header.

    • HeaderMode.NORMAL: Process the header in non-strict mode. If a schema column is missing, a null value is ingested for that schema column. Any file column whose name does not correspond to a schema column or a security label column is ignored.

    • HeaderMode.STRICT: Process the header in strict mode. The name of each header column should correspond to a schema column, a security label column, or be named IGNORE. Each schema column must appear in the file.

    Optional. Default=HeaderMode.NONE. Only applies to CSV files.

    New in version 1.11.0.

  • record_history (bool) –

    If true, records the history of the job.

    New in version 1.4.0.

  • row_labels (list) –

    A list of security labels to attach to each row inserted with the load. Each label must have been passed in to the row_label_universe parameter when creating the frame. Note: Only one of row_labels and row_label_columns must be passed.

    New in version 1.5.0.

  • row_label_columns (list) –

    A list of columns indicating which columns in the CSV file contain security labels to attach to the inserted row. If the header mode is NONE or IGNORE, this must be a list of integer column indices. If the header mode is NORMAL or STRICT, this must be a list of string column names. Note: Only one of row_labels and row_label_columns must be passed.

    New in version 1.5.0.

  • delimiter (str) –

    Delimiter for CSV data. Only applies to CSV files.

    New in version 1.5.1.

  • frame_to_input_column_mapping (dictionary) – Maps the frame column names to file columns for the ingest. The key of each element is a frame column name. The value is either the name of the file column (from the file header) or the file column index. If file column names are used, the header_mode must be NORMAL. If only file column indices are used, the header_mode can be NORMAL, NONE, or IGNORE.

  • suppress_errors (bool) –

    If true, continues to load data if an ingest error is encountered, placing the first 1000 errors into the job history. If false, stops on first error and raises. Defaults to False.

    New in version 1.11.0.

  • frame_to_file_column_mapping (dictionary) –

    Same as frame_to_input_column_mapping.

    Deprecated since version 1.13.0: Use frame_to_input_column_mapping instead.

  • input_filter (str) –

    TQL fragment used to filter, modify and parameterize the raw data from the input to produce the row data fed to the frame.

    New in version 1.14.0.

Returns:

A Job object representing the job that has executed the load.

Return type:

Job

Raises:

  • XgtIOError – If a file specified cannot be opened or if there are errors inserting any lines in the file into the frame.

  • XgtNameError – If the frame does not exist on the server.

  • XgtSecurityError – If the user does not have required permissions for this action.

  • XgtTransactionError – If a conflict with another transaction occurs.

property name

Name of the frame.

Type:

str

property num_rows

Gets the number of rows in the frame.

Type:

int

property row_label_universe

Gets the universe of row security labels that can be attached to rows of this frame. Only labels that are also in the authenticated user’s label set are returned.

Type:

list of str

save(path, offset=0, length=None, headers=False, record_history=True, include_row_labels=False, row_label_column_header=None, preserve_order=False, number_of_files=1, duration_as_interval=False, delimiter: str = ',')

Writes the rows from the frame to a file in the location indicated by the path parameter. Will save as a Parquet file if the extension is .parquet, otherwise saves as a CSV.

Parameters:

  • path (str) –

    Path to a file.

    Syntax for one file path

    Resource type

    Path syntax

    local to Python:

    ’<path to file>’ ‘xgt://<path to file>’

    xGT server:

    ’xgtd://<path to file>’

  • offset (int) – Position (index) of the first row to be retrieved. Optional. Default=0.

  • length (int) – Maximum number of rows to be retrieved. Optional. Default=None.

  • headers (boolean) – Indicates if headers should be added. Optional. Default=False.

  • record_history (bool) –

    If true, records the history of the job.

    New in version 1.4.0.

  • include_row_labels (bool) –

    Indicates whether the security labels for each row should be egested along with the row.

    New in version 1.5.0.

  • row_label_column_header (str) –

    The header column name to use for all row labels if include_row_labels is true and headers is true.

    New in version 1.5.0.

  • preserve_order (boolean) –

    Indicates if the output should keep the order the frame is stored in. Optional. Default=False.

    New in version 1.5.1.

  • number_of_files (int) –

    Number of files to save. Only works with the xgtd:// protocol. Optional. Default=1.

    New in version 1.10.0.

  • duration_as_interval (bool) –

    For Parquet files duration will be saved as the logical Interval type instead of the default 64 bit unsigned integer type. Only works with the xgtd:// protocol. Optional. Default=False.

    New in version 1.13.0.

  • delimiter (str) –

    Delimiter for saved CSV data. Only applies to CSV files.

    New in version 1.13.1.

Returns:

A Job object representing the job that has executed the save.

Return type:

Job

Raises:
property schema

Gets the property names and types of the frame.

Type:

list of lists