4. Memory and Compute Resources¶
4.1. Monitoring Usage¶
It is important to understand how much memory is required to hold the data graph(s) and other intermediate data resulting from computations.
To see the total memory in the server, a Python script can refer to the xgt.Connection.max_user_memory_size attribute.
import xgt
server = xgt.Connection()
print("Total Available Memory: {.3f}GiB".format(server.max_user_memory_size))
To see the total available (free) memory, a Python script can refer to the xgt.Connection.free_user_memory_size attribute.
import xgt
server = xgt.Connection()
print("Total Free Memory: {.3f}GiB".format(server.free_user_memory_size))
4.2. Setup and Configuration On-premises¶
The set-up of the memory and computing resources involves establishing limits on memory use, limits on use of CPUs, and how these memory and compute resources are to be used within the server platform. There are generally three ways of managing these resources:
A single instance of
xgtdrunning on the server platform that uses the whole system.A single instance of
xgtdrunning on the server platform that needs to share the platform with other software applications.Multiple instances of
xgtdsharing a server platform.
4.3. Using the Whole Server Platform¶
In this scenario, where the xGT server will use the entire machine, the memory assigned to the server application will be the smallest of:
the configured limit for
system.max_memorythe amount of available RAM on the server platform
If the system.max_memory is not configured, then xgtd will use the smallest of the other values.
If the server hardware has Non-Uniform Memory Access (NUMA), then the strategy on how to allocate data among the NUMA nodes can have a significant impact on performance.
The recommended NUMA placement is to evenly distribute data across all NUMA nodes.
This can be done with numactl when launching the server:
numactl --interleave=all /path/to/bin/xgtd
The default xgtd.service file for this scenario is:
[Unit]
Description=xgtd
After=multi-user.target
[Service]
Type=simple
ExecStart=/usr/bin/numactl --interleave=all -- /opt/xgtd/bin/xgtd -c /etc/xgtd/xgtd.conf -p 4367
User=xgtd
Environment=LD_LIBRARY_PATH=/opt/xgtd/lib
WorkingDirectory=/
[Install]
WantedBy=multi-user.target
This file, if placed in the /etc/systemd/system/xgtd.service location, will control the xgtd daemon via:
$ systemctl status xgtd$ systemctl start xgtd$ ststemctl stop xgtd
There are two components in this xgtd.service file that a site may consider changing:
If no configuration is provided for the
system.io_directorysetting, theWorkingDirectory=establishes the location the server will look for data on aload()operation and the location to create CSV files on asave()operation (see Configuring the Server).The parameters on the
ExecStart=statement. Some likely candidates are the program arguments for thenumactlprogram and the port number to listen on. Note that the port number may also be inside the/etc/xgtd/xgtd.conffile.
4.5. Running Multiple xgtd Instances¶
This scenario is very similar to the previous one where xgtd is sharing a platform with some other application.
The major difference is that the sharing is now between multiple instances of the xgtd server.
In order to accomplish this, it may be necessary to have multiple xgtd.conf files and multiple xgtd.service files.
We recommend placing the xGT configuration files in /etc (e.g., /etc/xgtd/xgtd.conf, /etc/xgtd/xgtd.production.conf).
The xgtd.service file is placed into /etc/systemd/system/ by the package installer.
Launching multiple instances of xgtd is best handled by replicating the xgtd.service file with unique names that have instance-specific information inside such as a port number.
Note that multiple xgtd.service files can use a shared xgtd.conf file if the only difference would be a port number, since the -p program argument can override the configured port.
A common setup is to have multiple xgtd.service files, each with different CPU sets and port numbers.
We show two sample service files that split a machine into two server daemons, one using 3/4 of the machine for production on port 4367 and the other using the other 1/4 of the machine for testing on port 4368.
# /etc/system/system/xgtd-production.service
[Unit]
Description=xgtd
After=multi-user.target
[Service]
Type=simple
ExecStart=/usr/bin/numactl --cpunodebind=8-31 --interleave=8-31 -- /opt/xgtd/bin/xgtd -c /etc/xgtd/xgtd.conf -p 4367
User=xgtd
Environment=LD_LIBRARY_PATH=/opt/xgtd/lib
WorkingDirectory=/
[Install]
WantedBy=multi-user.target
# /etc/system/system/xgtd-test.service
[Unit]
Description=xgtd
After=multi-user.target
[Service]
Type=simple
ExecStart=/usr/bin/numactl --cpunodebind=0-7 --interleave=0-7 -- /opt/xgtd/bin/xgtd -c /etc/xgtd/xgtd.conf -p 4368
User=xgtd
Environment=LD_LIBRARY_PATH=/opt/xgtd/lib
WorkingDirectory=/
[Install]
WantedBy=multi-user.target