6.1. An Introduction to Graph Analysis with xGT

6.1.1. Setting up the xGT Environment

For this demonstration, we’ll be running the xGT Server on an Amazon EC2 instance and interacting with it over a secure SSH tunnel from our laptop. More details and alternative methods for getting xGT running can be found at docs.trovares.com.

Connection Setup

Connection Setup

6.1.1.1. Install xgt on Our Local Machine

!pip install --quiet --upgrade xgt

6.1.1.2. Checking the xGT Server Is up

# Assume "cloud" is the name of the server platform
!ssh cloud ps -e | grep xgtd

6.1.1.3. Establishing a Secure Tunnel

# Assume "ec2-user" is a username on the server platform
!ssh -fNL 4367:localhost:4367 ec2-user@cloud
!ps -A | grep 'ssh[ ]-fNL'

6.1.2. Getting Started with xGT

First, let’s import the package and ensure our connection is good.

import xgt
conn = xgt.Connection()
conn
  <xgt.connection.Connection at 0x111d73c70>

6.1.3. Loading Data

We’ll be using the LANL Unified Host and Network Dataset, a set of netflow and host event data collected on an internal Los Alamos National Lab network.

Our goal will be to turn about 100GB of CSV files into a single connected graph.

6.1.3.1. Step 1: Create a Graph

Data Model

Data Model

Vertices: Los Alamos uses a free-form, anonymized string called a “device” as a host identifier (analogous to an IP address). We’ll use these as vertices in our graph:

[conn.drop_frame(_) for _ in ['lanl__Netflow','lanl__HostEvents','lanl__Devices']]
conn.drop_namespace('lanl')
conn.drop_namespace('results', force_drop=True)
  False
devices = conn.create_vertex_frame(
    name='lanl__Devices',
    schema=[['device', xgt.TEXT]],
    key='device')

devices
  <xgt.graph.VertexFrame at 0x111cf3c40>

Edges: The LANL dataset contains two types of data: netflow and windows log events. Of the log events recorded, some describe events within a host/device (e.g., reboots), and some describe authentication events that may be between devices (e.g., login from device A to B). We’ll call the authentication events AuthEvents and the others we’ll call HostEvents. In this notebook we load only the Netflow data and HostEvents.

netflow = conn.create_edge_frame(
    name='lanl__Netflow',
    schema=[['epoch_time', xgt.INT],
            ['duration', xgt.INT],
            ['src_device', xgt.TEXT],
            ['dst_device', xgt.TEXT],
            ['protocol', xgt.INT],
            ['src_port', xgt.INT],
            ['dst_port', xgt.INT],
            ['src_packets', xgt.INT],
            ['dst_packets', xgt.INT],
            ['src_bytes', xgt.INT],
            ['dst_bytes', xgt.INT]],
    source=devices,
    target=devices,
    source_key='src_device',
    target_key='dst_device')

netflow
  <xgt.graph.EdgeFrame at 0x111dbe070>
host_events = conn.create_edge_frame(
    name='lanl__HostEvents',
    schema=[['epoch_time', xgt.INT],
            ['event_id', xgt.INT],
            ['log_host', xgt.TEXT],
            ['user_name', xgt.TEXT],
            ['domain_name', xgt.TEXT],
            ['logon_id', xgt.INT],
            ['process_name', xgt.TEXT],
            ['process_id', xgt.INT],
            ['parent_process_name', xgt.TEXT],
            ['parent_process_id', xgt.INT]],
    source=devices,
    target=devices,
    source_key='log_host',
    target_key='log_host')

host_events
  <xgt.graph.EdgeFrame at 0x111dbe100>

6.1.3.2. Step 2: Load the Data

With all the data types described, we can actually add data fitting those descriptions directly from data files. We load these directly into the edge or vertex structures they correspond to.

# Utility to pretty-print the data currently in xGT
def print_data_summary():
  print('Devices (vertices): {:,}'.format(devices.num_vertices))
  print('Netflow (edges): {:,}'.format(netflow.num_edges))
  print('Host event (edges): {:,}'.format(host_events.num_edges))

print_data_summary()
  Devices (vertices): 0
  Netflow (edges): 0
  Host event (edges): 0

Load the 1-sided host event data:

%%time
urls = ["https://datasets.trovares.com/LANL/xgt/wls_day-02_1v.csv"]
host_events.load(urls)
print_data_summary()
  Devices (vertices): 10,086
  Netflow (edges): 0
  Host event (edges): 16,781,773
  CPU times: user 12.2 ms, sys: 15.6 ms, total: 27.9 ms
  Wall time: 48.3 s

Load the netflow data:

%%time
urls = ["https://datasets.trovares.com/LANL/xgt/nf_day-02.csv"]
netflow.load(urls)
print_data_summary()
  Devices (vertices): 31,324
  Netflow (edges): 115,949,436
  Host event (edges): 16,781,773
  CPU times: user 41.1 ms, sys: 59.3 ms, total: 100 ms
  Wall time: 3min 26s

6.1.4. Querying Our Graph

We’ll be looking for a mock pattern, similar to one that might be used to detect bot-net behavior. The pattern reflects an infected host (a) which is connecting up to a bot-net command and control node (b) with an exfiltration connection to a collection node (c).

Zombie Reboot

Zombie Reboot

  • Some device A boots up and, within a short amount of time, starts up a program.

  • Shortly afterwards, device A sends a message to some other device B.

  • Device B has a long-standing connection to another device C, which has been open for at least an hour, started before A booted, and remained open after A sent a message to B.

# Query helper function
import time
def run_query_and_count(query):
    conn.drop_frame('results__Answers')
    start_time = time.time()
    conn.wait_for_metrics()
    wait_time = time.time() - start_time
    if wait_time > 30:
      print('Time to wait for metrics: {:3,.2f}'.format(wait_time))
    conn.run_job(query)
    # Retrieve count
    table = conn.get_table_frame('results__Answers')
    count = table.get_data()[0][0]
    return count

6.1.4.1. Query 1: Search for Just a Boot Event

Normally, we might not know exactly which pattern to look for, and refining that pattern would be part of the analytic process. We’ll start by just finding devices with boot events.

Boot Only

Boot Only

%%time
q = """
MATCH (a)-[boot:lanl__HostEvents]->(a)
WHERE boot.event_id = 4608
RETURN count(*)
INTO results__Answers
"""
count = run_query_and_count(q)
print('Number of boot events: ' + '{:,}'.format(count))
  Number of boot events: 1,891
  CPU times: user 4.17 ms, sys: 2.2 ms, total: 10.4 ms
  Wall time: 148 ms

6.1.4.2. Query 2: Boot Event Followed by Program Start Event

Now we’ll refine the query by looking for programs launched within 4 seconds of a boot event.

Boot and Program Start

Boot and Program Start

%%time
q = """
MATCH (a)-[boot:lanl__HostEvents]->(a)-[program:lanl__HostEvents]->(a)
WHERE boot.event_id = 4608
  AND program.event_id = 4688
  AND program.epoch_time >= boot.epoch_time
  AND program.epoch_time - boot.epoch_time < 4
RETURN count(*)
INTO results__Answers
"""

count = run_query_and_count(q)
print('Number of boot & program start events: ' + '{:,}'.format(count))
  Number of boot & program start events: 547,759
  CPU times: user 5.58 ms, sys: 3.34 ms, total: 8.92 ms
  Wall time: 231 ms

6.1.4.3. Query 3: Boot, Program Start, and Connection

Now we add the network connection edge from A to B from port 3128 (in our mock example, this is a signature port for the bot in question). We’ll restrict it such that all the pieces (boot, program start, and message) happen within 4 seconds.

Boot, Program Start, and Connection

Boot, Program Start, and Connection

%%time
q = """
MATCH (a)-[boot:lanl__HostEvents]->(a)-[program:lanl__HostEvents]->(a)
         <-[nf1:lanl__Netflow]-(b)
WHERE a <> b
  AND nf1.src_port = 3128
  AND boot.event_id = 4608
  AND program.event_id = 4688
  AND program.epoch_time >= boot.epoch_time
  AND nf1.epoch_time >= program.epoch_time
  AND nf1.epoch_time - boot.epoch_time < 4
RETURN count(*)
INTO results__Answers
"""
# Note the overall time limit on the sequence of the three events

count = run_query_and_count(q)
print('Number of boot, programstart, & nf0 events: ' + '{:,}'.format(count))
  Number of boot, programstart, & nf0 events: 75
  CPU times: user 4.4 ms, sys: 2.34 ms, total: 6.73 ms
  Wall time: 531 ms

6.1.4.4. Query 4: Full Zombie Reboot Pattern

Finally, we add in the last network connection and match the full pattern.

Zombie Reboot

Zombie Reboot

%%time
q = """
MATCH (a)-[boot:lanl__HostEvents]->(a)-[program:lanl__HostEvents]->(a)
         <-[nf1:lanl__Netflow]-(b)-[nf2:lanl__Netflow]->(c)
WHERE a <> b AND b <> c AND a <> c
  AND nf1.src_port = 3128
  AND boot.event_id = 4608
  AND program.event_id = 4688
  AND program.epoch_time >= boot.epoch_time
  AND nf1.epoch_time >= program.epoch_time
  AND nf1.epoch_time - boot.epoch_time < 4
  AND nf2.duration >= 3600
  AND nf2.epoch_time < nf1.epoch_time
  AND nf2.epoch_time + nf2.duration >= nf1.epoch_time
RETURN count(*)
INTO results__Answers
"""

count = run_query_and_count(q)
print('Number of zombie reboot events: ' + '{:,}'.format(count))
  Number of zombie reboot events: 11,925
  CPU times: user 4.35 ms, sys: 2.66 ms, total: 7.01 ms
  Wall time: 1.11 s

Copyright © 2017-2020 Trovares Inc