xGT Release Notes

1.3.1 (02-28-2020)

Changed

• Improved the entitlement process.

 

---

 

1.3.0 (11-05-2019)

New Features

• Added the ability to compute metrics about frame data and build a cache of these metrics.
• Added a query planner that reorders queries to improve query performance. The planner explores many query plans and computes a cost-metric for each plan using the cached metrics. The least-cost plan is chosen for the actual running of the query.
• Turned on the metrics cache collection and query planning by default. Metrics cache collection may be turned off via an xgtd configuration variable. Query planning can be turned off by calling set_optimization_level().
• Added the Connection object functions wait_for_metrics() and get_metrics_status() to know synchronously or asynchronously when queries to fill the metrics cache have finished running.
• Added Cypher support for the string concatenation operator, +, and the toString() function.
• A space is now allowed as a separator of the date and time in a DATETIME in addition to T.
• Added the ability to collect the number of visited edges for a job. The collection is turned on and off via an xgtd configuration variable.
• Added DRM to the xgtd server to support on-premises installations.

Changed

• Updated the names and format of the xgtd configuration variables given in xgtd.conf. The new format uses multiple levels of JSON objects to group categories of variables.
• Changed error_frame_name to be a property instead of a method for all frame types.
• Changed the behavior when a transaction fails to now throw an error to the Python client from run_job() and wait_for_job(). The previous behavior was for the functions to complete successfully, and the user had to check the job status to know if the transaction failed.
• Added the job status rollback to indicate when a job failed because of a transactional conflict with another job.
• Improved transactional consistency when querying for frame existence or metadata information about frames.
• Improved the performance of queries over frames that have had data deleted.
• Improved the performance and memory usage of some of the aggregators and solution modifiers used in queries.

Fixed

• Improved the parsing of all input types to better enforce constraints on the types.
• Added a check to prevent null values for key columns in vertex and edge frames when inserting or loading data.
• Fixed a bug where two queries writing to the same results table could cause a segfault.
• Fixed a bug where creating an edge frame with vertex frames that already contained data wasn't handled properly.
• Fixed a bug that severely slowed downloads from URLs with libcurl versions < 7.38 which includes installations on CentOS 7.
• Fixed numerous bugs relating to scheduling and managing jobs.
• Fixed a segfault that occurred in the xgtd server in some cases when memory is exhausted instead of gracefully stopping the current operation and throwing an error to the user.
• Fixed a bug where query results could be added to a pre-existing results table even if the transaction failed.
• Fixed a bug where cancelling a job didn't always terminate the job execution on the server.
• Fixed a bug where sometimes frames weren't immediately removed from the server when dropped.
• Fixed a bug where the server wasn't always releasing file descriptors.
• Fixed a bug where using the same column for the source and target keys of an edge wasn't allowed.
• Fixed a bug where a floating point value of nan was encoded improperly when being transferred between client and server.
• Fixed a bug where the header line was being written twice when saving a CSV file of frame data local to the client.
• Fixed a bug where the unsupported Cypher fragment RETURN * caused a segfault. It now throws an error to the Python client.

Unresolved

• There is a small chance that queries to build the metrics cache could cause a transactional failure in a user query or cause a frame insert or load to fail.

 

---

 

1.2.0 (06-04-2019)

New

• Released the xGT Python interface into open source. The xgt package can now be found on the Python Package Index and installed using pip install xgt.
• Added support for the Cypher keyword SET for updating non-key properties in frames.
• Added support for the Cypher keyword CREATE for adding new vertices and edges to a frame.
• Added support for the Cypher keyword MERGE for adding new vertices to a frame if the vertex key doesn't already exist.
• Added support for the Cypher keyword DETACH DELETE for removing vertices and all their incident edges from a frame.
• Added support for the Cypher keyword DELETE for removing edges from a frame.
• Added a transactional model. All operations are now transactional and safe to be run concurrently.
• Added the functions max_user_memory_size() and free_user_memory_size() to Connection so a user can get the maximum and free memory available to them on the server.
• Added the function error_frame_name() to TableFrame, VertexFrame, and EdgeFrame that returns the name used for the error table frame generated when there are errors on inserting data into a frame.
• Added the .xgtd.conf configuration variable io_threads that limits the number of threads used to read a single input file to improve performance on systems with large thread counts.

Changed

• Improved handling of out-of-memory situations so that in many cases a transaction that causes an out-of-memory error performs a rollback and leaves the system in a usable state.
• Improved error types and messages returned to the user.
• Improved error detection for vertex keys to catch duplicates earlier.
• Added a check for unsupported Cypher keywords and throw errors when they are used.
• Added a check that the columns in a RETURN statement match the schema of a pre-existing results table and throw an error when they don't.
• Added a check that the Cypher functions outdegree() and indegree() aren't being called on an edge and throw an error when they are.
• Removed unneeded files from the xgt library distribution package.
• Improved performance of downloading frame data from the server to the client.
• Improved performance of inserting data into frames.
• Added user documentation for all new features.

Fixed

• Fixed a bug where the server wasn't responsive to commands when a query was running.
• Fixed a bug where using the Cypher keyword STARTS WITH on a null property incorrectly threw an exception.
• Fixed a bug where a groupby on multiple Cypher outdegree() function calls yielded incorrect results.
• Fixed a bug that occasionally occurred when using ORDER BY in a query.
• Fixed a bug where downloading frame data was limited to 4MB.
• Fixed a bug on reconnection that occurred when a server unexpectedly exits during a script.
• The minimum version of the grpcio python package required by xgt is increased to 1.20.
• Python2 versions below 2.7.10 are now deprecated. Added a deprecation warning for a too low version of python at xgt module import time.

 

---

 

1.1.0 (02-21-2019)

New

• Added support for an SSL-encrypted gRPC connection between client and server.

Changed

• Removed the REST API from the Python library, server, and documentation.
• Removed the option to run in 'local' (unixsocket) mode as that security need has been replaced by the SSL-encrypted gRPC connection.
• Added the ability for the client to detect when the server has restarted and notify the user they need to reestablish a connection.
• Improved the hierarchical organization of exception messages.
• Improved the text of many errors returned to the Python user.
• Improved load and insert error reporting to return a usable frame with the valid data and an error table with all the invalid data and errors instead of throwing on the first error detected in the data.
• Improved logging of out-of-memory errors.
• Added error reporting when a Cypher statement compares a vertex with an edge.
• Added documentation for xGT data types.
• Added compiler optimizations to improve performance of some queries.
• Improved the client-server connection to reduce latency.
• Improved ingestion performance for large edge frames when implicitly filling the vertex frame.

Fixed

• Fixed numerous bugs where Null values weren't handled correctly or were handled differently from Cypher's behavior.
• Fixed numerous bugs where datatype conversion wasn't being handled correctly.
• Fixed a bug where xGT ran out of memory when ingesting large edge frames due to fragmentation when plenty of memory was available.
• Fixed a deadlock that could occur during load or insert operations.
• Fixed some client-server connection bugs when a Jupyter Notebook is used as the client.

 

---

 

1.0.0 (12-01-2018)

New

• Added ability to query a TableFrame using syntax for querying a VertexFrame.
• Names can now contain Unicode characters.

Changed

• Improved stability and performance of connection between Python client and xgtd server by switching to using gRPC instead of REST as the default communication protocol. The REST protocol is now deprecated.
• Added check when creating VertexFrame that the key parameter is given and is a valid column in the schema.
• Added check when creating EdgeFrame that the source_key and target_key parameters are given and are valid columns in the schema.
• Modified behavior when dropping frames to prevent a user from dropping a frame on which other frames depend.
• Modified behavior when creating frames to require a name for the frame.
• Disallowed the creation of frames with names containing periods.
• Renamed xgtd configuration file from .gemsconfig to .xgtd.conf, and now look for it in /etc if the file doesn't exist in the user's home directory.
• In xgtd.conf, renamed the property 's3_key_id' to 'aws_access_key_id' and 's3_key' to 'aws_secret_access_key' to be consistent with the naming in the aws credentials file.
• Errors are now reported when .xgtd.conf contains invalid property names or values of the wrong type.
• Modified exceptions to use new naming scheme.
• Improved handling of filename paths passed to I/O commands to make all filename paths unambiguous as to the location of the file.
• Added support for loading CSV files where some of the lines have more columns than the required number.
• Improved documentation and tutorials.

Fixed

• Fixed a rare bug where simultaneously creating and dropping frames caused an error.
• Fixed a bug where frames with numeric names couldn't be dropped.
• Fixed a bug where using a list of tuples as the schema to create a frame had stopped working.
• Fixed a bug where exceptions were sometimes reported to the wrong job when multiple jobs were running simultaneously.
• Fixed numerous bugs that occurred when loading data.

 

---

 

0.18.0 (10-24-2018)

Changed

• Modified the API for creating and destroying graphs. See the Python API documentation for details.
• Simplified the specification of source and target keys by supporting only a single key column.
• Names now have to be unique across all objects vs. unique only for a particular type of object.
• Changed working directory of AWS marketplace daemon to /srv/xgtd/data from /home/ec2-user/.

Fixed

• Improved type checking in the query compiler to provide better error messages to users for invalid queries.
• Improved error messages in some cases to make them more understandable.
• Fixed a bug where exceptions were sometimes reported to the wrong job when multiple jobs were running simultaneously.
• Fixed a bug where exceptions sometimes had malformed JSON.
• Fixed a segfault that occasionally occurred in the job manager.

 

---

 

0.17.0 (10-01-2018)

Changed

• Added additional error messages.
• Added query annotations to more types of invalid Cypher query error messages.
• Can now read AWS credentials directly from .aws/credentials.

Fixed

• Fixed some Unicode bugs.
• Fixed a segfault occurring when an exception is thrown while ingesting.

 

---

 

0.16.0 (09-01-2018)

Changed

• The query compiler can now automatically infer the vertex or edge frame when there is a single type.
• Standardized error message handling between run_job() and wait_for_jobs().

Fixed

• Improved compatibility between Python 2 and 3.
• Fixed spurious warnings about numpy datatype incompatibilities.
• Improved error reporting to give more meaningful error messages.
• Fixed many cases where errors were not being reported to the user.
• Fixed numerous bugs.