altimeter.core.neptune package

Submodules

altimeter.core.neptune.client module

Client for loading and accessing Altimeter generated data in Neptune

class AltimeterNeptuneClient(max_age_min, neptune_endpoint)

Bases: object

Client to run sparql queries against a neptune instance using graph name conventions to determine most recent graph.

Parameters

  • max_age_min (int) – maximum acceptable age in minutes of graphs. Only graphs which are found that meet this critera will be queried.

  • neptune_endpoint (NeptuneEndpoint) – NeptuneEndpoint object for this client

clear_graph_data(uri)

Clear a graph in Neptune

Return type

None

clear_graph_metadata(name, uri)

Clear a graph metadata entry

Return type

None

clear_registered_graph(name, uri)

Remove data and metadata for a graph by uri

Parameters

  • name (str) – graph name

  • uri (str) – graph uri

Raises

NeptuneUpdateGraphException if an error occurred during clearing –

Return type

None

connect_to_gremlin()

Get the Gremlin traversal and connection for the Neptune endpoint :rtype: Tuple[traversal, DriverRemoteConnection] :return: The Traversal object

get_graph_metadatas(name, version=None)

Return all graph metadatas for a given name/version. These represent fully loaded graphs in the Neptune database.

Parameters

  • name (str) – graph name

  • version (Optional[str]) – graph version

Return type

List[GraphMetadata]

Returns

list of GraphMetadata objects for the given graph name/version

get_graph_uris(name)

Return all graph uris regardless of whether they have corresponding metadata entries

Parameters

name (str) – graph name

Return type

List[str]

Returns

list of graph uris

load_graph(bucket, key, load_iam_role_arn)

Load a graph into Neptune. :type bucket: str :param bucket: s3 bucket of graph rdf :type key: str :param key: s3 key of graph rdf :type load_iam_role_arn: str :param load_iam_role_arn: arn of iam role used to load the graph

Return type

GraphMetadata

Returns

GraphMetadata object describing loaded graph

Raises

NeptuneLoadGraphException if errors occur during graph load –

static parse_arn(arn)

Parses an ARN into the component pieces :type arn: str :param arn: The arn to parse :rtype: Dict :return: A dictionary of the arn pieces

prepare_request(method='GET', payload='', querystring={})

This prepares the request for sigv4signing. This is heavily influenced by the code here: https://github.com/awslabs/amazon-neptune-tools/tree/master/neptune-python-utils :type method: str :param method: The method name :type payload: str :param payload: The request payload :type querystring: Dict :param querystring: The request querystring :rtype: RequestParameters :return: The request parameters

run_query(graph_names, query)

Runs a SPARQL query against the latest available graphs given a list of graph names.

Parameters

  • graph_names (Set[str]) – list of graph names to query

  • query (str) – query string. This query string should not include any ‘from’ clause; the graph_names param will be used to inject the correct graph uris by locating the latest acceptable (based on max_age_min) graph.

Return type

QueryResult

Returns

QueryResult object

run_raw_query(query)

Run a query against a neptune instance, return a dict of results. Generally this should be called from run_query

Parameters

query (str) – complete query to run

Return type

QueryResultSet

Returns

QueryResultSet object

Raises

NeptuneQueryException if an error occurred running the query –

write_to_neptune_lpg(graph, scan_id)

Writes the graph to a labeled property graph :type scan_id: str :param scan_id: The unique string representing the scan :type graph: Dict :param graph: The graph to write :rtype: None :return: None

write_to_neptune_rdf(graph)

Writes the graph to an RDF graph :type graph: Dict :param graph: The graph to write :rtype: None :return: None

class GraphMetadata(uri, name, version, start_time, end_time)

Bases: object

A GraphMetadata represents the details of a graph. These details are stored in a metadata graph in Neptune and used by clients to find the latest available successfuly loaded graph.

Parameters

  • uri (str) – graph uri

  • name (str) – graph name

  • version (str) – graph version

  • start_time (int) – epoch timestamp of graph start time

  • end_time (int) – epoch timestamp of graph end time

end_time: int
name: str
start_time: int
uri: str
version: str
class NeptuneEndpoint(host, port, region, ssl=True, auth_mode='DEFAULT')

Bases: object

Represents an AWS Neptune endpoint.

Parameters

  • host (str) – neptune host

  • port (int) – neptune port

  • region (str) – neptune region

auth_mode: str = 'DEFAULT'
get_endpoint_str()

Get the endpoint as a string in host:port format

Return type

str

Returns

endpoint string

get_gremlin_endpoint(ssl=True)

Get the loader url for this Neptune endpoint

Return type

str

Returns

loader endpoint url

get_loader_endpoint(ssl=True)

Get the loader url for this Neptune endpoint

Return type

str

Returns

loader endpoint url

get_sparql_endpoint(ssl=True)

Get the sparql query url for this Neptune endpoint

Return type

str

Returns

sparql query endpoint url

host: str
port: int
region: str
ssl: bool = True
class RequestParameters(uri, querystring, headers)

Bases: object

Holds the request parameters for Sigv4Signing

discover_neptune_endpoint()

Find a Neptune

Return type

NeptuneEndpoint

get_required_tag_value(tag_set, key)

Get a tag value from a TagSet. Raise ValueError if the key is not present.

Parameters

  • tag_set (List[Dict[str, str]]) – list of dicts, each of which contains keys ‘Key’ and ‘Value’.

  • key (str) – tag key string

Return type

str

Returns

tag value string

Raises

ValueError if key is not present in tag_set –

altimeter.core.neptune.exceptions module

Neptune errors

exception InvalidQueryException

Bases: altimeter.core.neptune.exceptions.NeptuneClientException

A statically detected error with a query was found.

exception NeptuneClearGraphException

Bases: altimeter.core.neptune.exceptions.NeptuneClientException

An error occurred while clearing a graph.

exception NeptuneClientException

Bases: altimeter.core.exceptions.AltimeterException

Base exception class for Neptune client exceptions.

exception NeptuneLoadGraphException

Bases: altimeter.core.neptune.exceptions.NeptuneClientException

An error occurred while loading a graph.

exception NeptuneNoFreshGraphFoundException

Bases: altimeter.core.neptune.exceptions.NeptuneClientException

No acceptably recent graph could be found in Neptune.

exception NeptuneNoGraphsFoundException

Bases: altimeter.core.neptune.exceptions.NeptuneClientException

No graphs were found in Neptune.

exception NeptuneQueryException

Bases: altimeter.core.neptune.exceptions.NeptuneClientException

A server-side error occurred during a Neptune query execution.

exception NeptuneUpdateGraphException

Bases: altimeter.core.neptune.exceptions.NeptuneClientException

An error occurred while updating a graph.

altimeter.core.neptune.results module

Classes for representing query results

class QueryResult(graph_uris_load_times, query_result_set)

Bases: object

Represents the results of a SPARQL query and includes the graph uris from which results were pulled.

Parameters

  • graph_uris_load_times (Dict[str, int]) – Dict with keys which are the graph uris which were used in this query and values which are the load end times for the graph.

  • query_result_set (QueryResultSet) – QueryResultSet containing results

get_length()

Get the length of this result.

Return type

int

Returns

int length

get_stats(field_keys)

Return a Counter representing statistics about this result set keyed by a user specified list of field keys (e.g. account_id and account_name)

Parameters

field_keys (List[str]) – list of field names to use as stat keys

Return type

Counter

Returns

Counter containing result stats.

to_csv()

Create a CSV representation of this QueryResult.

Return type

str

Returns

csv as a str

to_dict()

Generate a dict representing this QueryResult

Return type

Dict[str, Union[List[Any], Dict[str, int]]]

Returns

dict representation of this QueryResult

to_list()

Generate a list representing this QueryResult

Return type

List[Dict[str, Any]]

Returns

List of dicts representing this QueryResult

to_ndjson()

Create an NDJSON representation of this QueryResult.

Return type

str

Returns

NDJSON as a str

class QueryResultSet(fields, values)

Bases: object

Represents the results of a SPARQL query.

Parameters

  • fields (List[str]) – List of field names

  • values (List[Dict[str, Any]]) – list of value dicts as returned from neptune’s query api.

classmethod from_dict(data)
Return type

QueryResultSet

classmethod from_sparql_endpoint_json(resp)

Build a QueryResultSet object from the returned data of a sparql endpoint json query (has top level field ‘head’ and results’)

Parameters

resp (Dict[str, Any]) – response dict from neptune’s query api

Return type

QueryResultSet

Returns

QueryResultSet object

get_stats(field_keys)

Return a Counter representing statistics about this result set keyed by a user specified list of field keys (e.g. account_id and account_name)

Parameters

field_keys (List[str]) – list of field names to use as stat keys

Return type

Counter

Returns

Counter containing result stats.

to_csv()

Create a CSV representation of this QueryResultSet.

Return type

str

Returns

csv as a str

to_list()

Create a list of dicts representing these results, each dict is an individual result row.

Return type

List[Dict[str, Any]]

Returns

List of dicts representing this QueryResultSet.

to_ndjson()

Create an NDJSON representation of this QueryResult.

Return type

str

Returns

NDJSON as a str

altimeter.core.neptune.sparql module

SPARQL related functions.

finalize_query(query, graph_uris)

Finalize a generic sparql query - specifically add a FROM clause containing graph uris for this query.

Parameters

  • query (str) – query string

  • graph_uris (List[str]) – list of graph uris

Return type

str

Returns

finalized query string

Module contents