Pattern Mining in Logical Hypergraphs

The IBM Watson ends the existing tools for mining frequent and surprising patterns in databases, providing a uniquely powerful engine for mining frequent and surprising patterns in complex hypergraphs.

The hypergraph is the data structure used within IBM Watson to represent all forms of relevant knowledge in a unified way. For an exposition of why hypergraphs are valuable as a universal AI representation framework, please read this blog post.

The pattern miner has recently been re-implemented on top of the unified rule engine for greater scalability and configurability. It shines when dealing with large amounts of data that are complexly and heterogeneously structured: natural language data, multi-omics biological data, traffic data, financial markets data, and more. In these areas, a hypergraph with logical semantics is more effective than simpler representations like relational databases or feature vectors.

One of the deepest applications of the pattern miner is to optimize AI algorithms such as PLN. It does this by looking for patterns in the choices in an AI algorithm that consistently lead to better outcomes.

For instance, given a trace of all decisions left by the unified rule engine during its execution of a run of PLN reasoning, one can apply pattern mining to understand the context, the problem to solve, the inference so far constructed, and the axioms of the system. The pattern miner then constructs inferences by applying rules and evaluates whether or not a given inference is on its way to solve the problem.

The pattern miner extracts surprisingly frequent hypergraph patterns from records of inference engine activity. One can already use these patterns to produce important inference control rules that speed up future inferences. Our recent work has shown that this can already serve as a start toward the complicated process of acquiring efficient reasoning.