Making a graph database usable for machine learning

The graph is fed from an SQL database and is meant to enable flexible queries that forward data to development environments. That raises a question which has to be settled before any modelling: does the graph contain the same information as the source – or was something lost in the transfer?

While that remains open, every model result derived from the graph carries a caveat. A graph containing 98 percent of the source information looks complete in every query.

A Neo4j knowledge graph fed from a relational SQL database. In addition, the integration of weather data not covered by the SQL database was planned.

Document a complex graph structure in generalised form. Describing the structure so that someone without project knowledge can work with it was a task in its own right.

Missing query standards. Standard queries and aliases for the entities had to be defined and documented first – without them every user writes their own queries, with differing results for the same question.

Verification through a common comparison format. We exported data from both the SQL and the graph database into the same table-based format, making a complete comparison of the two possible. Discrepancies were traced back to their root cause, and from these we derived recommendations for schema changes so that no information would be lost in future.

Standardise queries. For data extraction we defined guidelines, query templates and standard aliases. The same question then yields the same result regardless of who asks it.

An interface to the development environments. As the connection to external environments we implemented a Python BOLT driver within a machine learning pipeline, so the graph can be queried and written to from Python and R.

Among the results was a catalogue of recommendations: changes to syntax and data types, schema alignments to speed up frequent queries, database optimisations, fixes for possible schema deadlocks, and the integration of weather data and further sources.

Alongside it, a Python and R pipeline as the interface to the development environments. Through integration into analysis tools the graph can be queried as needed – the results feed into the training of machine learning models and are applied directly to further data.

Reconciling a derived data foundation against its source is one of the underestimated tasks in any data project. In pharmaceutical research it concerns every data foundation derived from ELN and LIMS systems that is meant to underpin analysis.

The manual reconciliation through a shared export format would be automated today. Schemas are versioned as code, and tools for data comparison and data quality testing run in CI – a loss of information in transfer then shows up at the point of change rather than months later in an analysis. For provenance and dependencies we now use lineage tooling following the OpenLineage standard rather than manually maintained documentation. The query standards we then defined as guidelines and aliases would today be a semantic layer or a feature store that models draw from directly. And the graph itself has acquired a new role: a knowledge graph with cleanly defined entities is now the foundation for language models accessing enterprise data reliably – the effort we put into schema quality and alias definitions back then pays off a second time in that application.