9.1. Using xGT on AWS

The xGT server runs as a Linux daemon on an AWS instance you spin up and listens on a local port for commands. The server is driven by a Python client, which can be run either locally on the AWS instance or any other client by installing the xgt package.

9.1.1. Modes of Operation

There are three general modes of operation for working with xGT on AWS.

9.1.1.1. Local to AWS

All Python scripts are run locally within the same AWS instance.

This strategy involves:

  • Logging in to your instance.

  • Bringing your own data to the instance.

  • Running scripts from any shell on the instance.

9.1.1.2. SSH Tunneling

All Python scripts are run on a remote system such as your laptop or desktop that is connected using an SSH Tunnel.

  • First, create an encrypted ssh tunnel. On an openssh-based platform (most UNIX-based systems) this can be done on the command line:

    $ ssh -i <private key file> -N -L 4367:localhost:4367 ec2-user@<instance DNS name or IP address>

  • Tunneling from a windows platform can be done using a third party package such as PuTTY using the provided graphical interface.

  • Users must install the xgt Python package on their client machine.

  • Once tunneling is set up, run your Python scripts on the client machine.

9.1.1.3. Using an Open Port

Warning

This method is insecure without enabling SSL support.

This is similar to the previous scenario but the EC2 instance has a port opened up for clients to connect. The default port for xgt and the one used on AWS is 4367.

  • Configure your EC2 instance to open port 4367.

  • The client machine running Python scripts will connect to the remote server at the open port.

Connecting to a remote server using the xgt package is straightforward:

>>> import xgt
>>> conn = xgt.Connection("<DNS name or IP address>")

Note that because this mode allows for data between client and server to be sent unencrypted, it is insecure and should really only be used in development. xGT supports SSL authentication for both client and server, which both encrypts all traffic back and forth as well as verifying client and server identities. Enabling SSL requires adding certificate files and restarting the server. More information can be found here: Using an SSL Secure Channel.

9.1.2. Running the xgtd Server Daemon

The server component of the xGT application is called xgtd. It is a daemon process launched automatically at boot. You can examine and control the xGT server using standard linux utilities:

sudo systemctl status xgtd
sudo systemctl stop xgtd
sudo systemctl start xgtd
sudo systemctl restart xgtd

9.1.2.1. Logging

When running xgtd using the using the strategy described above, the events are written into a log file named /var/log/xgtd.log.

9.1.2.2. Data Considerations

When using the xgtd:// protocol for reading and writing data, xGT will look for relative pathnames in the /srv/xgtd/data directory. For example, xGT would look for the path xgtd://folder/file in /srv/xgtd/data/folder/file.

To learn more about loading and saving, read Data Movement.

9.1.3. S3 Credentials

When loading a file using the load() method and an s3: protocol prefix, xGT will check the /home/xgtd/.aws/credentials file for the two variables aws_access_key_id and aws_secret_access_key. If the credentials file doesn’t exist, xgt will check for the configuration variables aws.access_key_id and aws.secret_access_key. The online AWS Access Keys document explains what these keys are and contains references to learn how to create and manage them. (These values may also be specified at runtime in user code by passing them to a Connection object.)

When reading the /home/xgtd/.aws/credentials file, xGT also supports profile selection via the AWS_PROFILE environment variable. If no environment variable is found, xGT will use the default profile.