2. Data Analytics

The following sections are available in this chapter:

• 2.1. Graph Thinking
  • 2.1.1. Introduction
  • 2.1.2. Graph Model
  • 2.1.3. Graph Image
  • 2.1.4. Setting up the Graph Model in xGT
    • 2.1.4.1. Understanding the Vertices
    • 2.1.4.2. Understanding the Edges
    • 2.1.4.3. Data Loading
  • 2.1.5. Looking for Interesting Patterns
    • 2.1.5.1. Describing the First Relationship
    • 2.1.5.2. Describing the Second Relationship
    • 2.1.5.3. Understanding the Query Result
    • 2.1.5.4. Exploring the Query Result
  • 2.1.6. Concluding Remarks
  • 2.1.7. Graph Thinking Jupyter Notebook
• 2.2. Frames and Namespaces
  • 2.2.1. Types of Frames
  • 2.2.2. Namespaces
    • 2.2.2.1. Default Namespace
    • 2.2.2.2. Disabling the Default Namespace
  • 2.2.3. Access Control
    • 2.2.3.1. Frame Access Control
    • 2.2.3.2. Row Access Control
• 2.3. Data Types and Operators
  • 2.3.1. Scalar Data Types
    • 2.3.1.1. BOOLEAN
    • 2.3.1.2. INT
    • 2.3.1.3. UINT
    • 2.3.1.4. FLOAT
    • 2.3.1.5. DATE
    • 2.3.1.6. TIME
    • 2.3.1.7. DATETIME
    • 2.3.1.8. DURATION
    • 2.3.1.9. IPADDRESS
    • 2.3.1.10. TEXT
  • 2.3.2. Composite Data Types
    • 2.3.2.1. LIST
  • 2.3.3. Operators
    • 2.3.3.1. Boolean Operators
    • 2.3.3.2. Comparison Operators
    • 2.3.3.3. Arithmetic Operators
    • 2.3.3.4. Temporal Operators
    • 2.3.3.5. String Operators
    • 2.3.3.6. List Operators
• 2.4. Frame Management
  • 2.4.1. Direct Frame Creation
    • 2.4.1.1. Example
    • 2.4.1.2. List Columns
  • 2.4.2. Implicit Frame Creation
    • 2.4.2.1. Implicit Vertex Frame Creation
    • 2.4.2.2. Implicit Edge Frame Creation
    • 2.4.2.3. Implicit Table Frame Creation
    • 2.4.2.4. Schema Inference
    • 2.4.2.5. Supported Data Sources
  • 2.4.3. Retrieving Frames
  • 2.4.4. Dropping Frames
  • 2.4.5. Modifying Frame Columns
    • 2.4.5.1. Appending Columns
    • 2.4.5.2. Deleting Columns
    • 2.4.5.3. Modifying All Columns
• 2.5. Data Movement
  • 2.5.1. Getting Data into xGT
    • 2.5.1.1. Loading from the Client Filesystem
    • 2.5.1.2. Loading from the Server Filesystem
    • 2.5.1.3. Loading from a URL
    • 2.5.1.4. Loading from an AWS S3 Bucket
    • 2.5.1.5. Inserting from the Client
    • 2.5.1.6. Loading CSV Files
    • 2.5.1.7. Loading Parquet Files
    • 2.5.1.8. Encoding Text to Types
    • 2.5.1.9. Reading into Edge Frames
    • 2.5.1.10. Reading into Vertex Frames
    • 2.5.1.11. Customizing Data Read Into Frames via TQL Input Filters
    • 2.5.1.12. Setting Row Labels
    • 2.5.1.13. Handling Ingest Errors
    • 2.5.1.14. Digging Deeper into Ingest Errors
  • 2.5.2. Getting Data out of xGT
    • 2.5.2.1. Saving to the Client Filesystem
    • 2.5.2.2. Saving to the Server Filesystem
    • 2.5.2.3. Saving to the Client
    • 2.5.2.4. Saving Parquet Files
    • 2.5.2.5. Saving Row Labels
  • 2.5.3. Performance Considerations
  • 2.5.4. S3 Credentials
  • 2.5.5. Arrow Flight Server Endpoint
• 2.6. Job Management
  • 2.6.1. Job Object
  • 2.6.2. Storing Temporary Query Results in Jobs
  • 2.6.3. Job History
• 2.7. The Trovares Query Language (TQL)
• 2.8. TQL Structure
  • 2.8.1. Putting It All Together
  • 2.8.2. Non-linear Patterns
  • 2.8.3. Variable-length Edge Traversal
  • 2.8.4. Multiple Edge Frame Traversal
  • 2.8.5. Path Variable Capture
• 2.9. Constraints
  • 2.9.1. Pattern Filters
  • 2.9.2. Functions
    • 2.9.2.1. Aggregation Functions
    • 2.9.2.2. Conversion Functions
    • 2.9.2.3. Mathematical Functions
    • 2.9.2.4. String Functions
    • 2.9.2.5. List Functions
    • 2.9.2.6. Path Functions
    • 2.9.2.7. Degree Functions
    • 2.9.2.8. Distinct Vertices Function
    • 2.9.2.9. Type Construction Functions
  • 2.9.3. Temporal Components
  • 2.9.4. CASE Expressions
    • 2.9.4.1. Simple CASE Expressions
    • 2.9.4.2. Generic CASE Expressions
    • 2.9.4.3. Examples
  • 2.9.5. Cypher Parameters
• 2.10. Property Extraction from Multiple Candidate Frames
  • 2.10.1. Limitations on Multiple Frame Property Extraction
• 2.11. List Clauses / Expressions
  • 2.11.1. UNWIND Clause
  • 2.11.2. List Comprehension
• 2.12. CALL Clause
  • 2.12.1. Whole Graph Algorithms
    • 2.12.1.1. Vertex and Edge Representation
    • 2.12.1.2. Breadth First Search
    • 2.12.1.3. PageRank
    • 2.12.1.4. Strongly Connected Components
    • 2.12.1.5. Weakly Connected Components
  • 2.12.2. Executing Algorithms on a Filtered Graph
    • 2.12.2.1. Performance Considerations
• 2.13. Modifications, Additions and Deletions
  • 2.13.1. Property Modifications
  • 2.13.2. Topology Additions
  • 2.13.3. Topology Deletions
  • 2.13.4. Row Additions and Deletions from Table Frames
  • 2.13.5. Visibility of Changes
• 2.14. Query Results
  • 2.14.1. Describing the Query Results
  • 2.14.2. Modifying the Results
  • 2.14.3. Where the Results Are Placed
    • 2.14.3.1. In a Named Frame
    • 2.14.3.2. In a Job
  • 2.14.4. Querying a Table Frame
  • 2.14.5. Multiple Query Parts
    • 2.14.5.1. Carrying Information Using WITH
    • 2.14.5.2. Aliasing in the WITH Clause
    • 2.14.5.3. Additional Filtering of WITH Partial Results
    • 2.14.5.4. Visibility of Changes
    • 2.14.5.5. Use Cases of Multiple Query Parts
    • 2.14.5.6. Additional Rules
    • 2.14.5.7. Performance Considerations
  • 2.14.6. Union Subqueries
    • 2.14.6.1. Visibility of Changes
  • 2.14.7. Optional Matches
    • 2.14.7.1. Valid Optional Structure
    • 2.14.7.2. Constraints
    • 2.14.7.3. Multiple Optional Matches
    • 2.14.7.4. Carrying Information Using WITH
  • 2.14.8. Returning Path Variables
• 2.15. Row Access Control in Queries
• 2.16. TQL for Cypher Users
  • 2.16.1. TQL Queries
    • 2.16.1.1. TQL is Typed
    • 2.16.1.2. Pattern Matching
    • 2.16.1.3. Query Structure
    • 2.16.1.4. Type Inferencing
    • 2.16.1.5. Pattern Variables
    • 2.16.1.6. Property and Topology Modifications
    • 2.16.1.7. Aggregation Functions and Solution Modifiers
  • 2.16.2. Query Examples
    • 2.16.2.1. Example 1
    • 2.16.2.2. Example 2
    • 2.16.2.3. Example 3
    • 2.16.2.4. Example 4
    • 2.16.2.5. Example 5
    • 2.16.2.6. Example 6
    • 2.16.2.7. Example 7
    • 2.16.2.8. Example 8
    • 2.16.2.9. Example 9
    • 2.16.2.10. Example 10
    • 2.16.2.11. Example 11
    • 2.16.2.12. Example 12
  • 2.16.3. Cypher Limitations
• 2.17. Access Control
  • 2.17.1. Security Labels
  • 2.17.2. Frame Access Control
    • 2.17.2.1. Creating Frames with Frame Security
    • 2.17.2.2. Namespace Labels
    • 2.17.2.3. Access Types for xGT Operations
  • 2.17.3. Row Access Control
    • 2.17.3.1. Attaching Row Labels to Data
    • 2.17.3.2. Security Labels Created by TQL Queries
    • 2.17.3.3. Performance
    • 2.17.3.4. Result Frames with Row Labels
    • 2.17.3.5. Viewing Security Labels
  • 2.17.4. Additional Rules
    • 2.17.4.1. Edge Access
    • 2.17.4.2. Degree Computation
    • 2.17.4.3. Row Label Universe Requirements
• 2.18. Transactions
  • 2.18.1. Transaction Basics
  • 2.18.2. Transactions in Python
  • 2.18.3. Loading and Saving Data
  • 2.18.4. Modifying Data in Queries
  • 2.18.5. Background Statistics Collection
  • 2.18.6. Implicit Frame Creation
• 2.19. Query Optimization
  • 2.19.1. Checking the Status of Statistics Collection
• 2.20. TQL Fragments
  • 2.20.1. Input TQL Fragments
  • 2.20.2. Output TQL Fragments
  • 2.20.3. Bound Variables
  • 2.20.4. Interaction with Different Data Sources
    • 2.20.4.1. Parquet Files
    • 2.20.4.2. CSV Files
    • 2.20.4.3. Python Data
  • 2.20.5. Requirements for Output Columns
  • 2.20.6. Putting It All Together
  • 2.20.7. Interaction with Different Data Targets
• 2.21. Frame Row References (Row IDs)
  • 2.21.1. Row IDs in Python
  • 2.21.2. Lifetime and Validity of Row IDs