TODO: Perform OWL Reasoner Evaluation

[callahantiff]

TODO: Perform comparison of OWL reasoners.

During today's meeting with @bill-baumgartner, we outlined how we will perform an evaluation of OWL reasoners on the PheKnowLator V.2.0 KG.

OWL Reasoner Selection Criteria
Using the following reviews (shown below), we selected reasoners that met the following criteria:

1. Low response time
2. Available via the OWLAPI
3. Open source

Khamparia A, Pandey B. Comprehensive analysis of semantic
web reasoners and tools: a survey. Education and Information
Technologies. 2017 Nov 1;22(6):3121-45.

Parsia B, Matentzoglu N, Gonçalves RS, Glimm B, Steigmiller A.
The OWL reasoner evaluation (ORE) 2015 competition report.
Journal of Automated Reasoning. 2017 Dec 1;59(4):455-82.

Eligible Reasoners:

Reasoner	Language	OWLTools
ELK	EL	Yes
ELepHant	EL	No
Pellet	DL	Yes
RACER	DL	No
FACT++	DL	No
Chainsaw	DL	No
Konclude	DL	No
Crack	DL	No
TrOWL	DL+EL	No
MORe	DL+EL	No

@bill-baumgartner - to determine what reasoners were available in OWLTools, I ran the following:

./owltools -h

---

Next Steps:

• Verify which of the reasoners above are currently included in OWLTools. For those that are not in the list, @bill-baumgartner and I will look into what is involved with adding the missing algorithms to OWLTools.

---

Evaluation Steps:

1. Benchmark each of the algorithms on HPO+Imports

   • Run-time
   • Justifications
   • Count of inferred axioms
   • Consistency

2. For all algorithms that pass the benchmark, run them against PheKnowLator

   • Including disjointness axioms
   • Excluding disjointness axioms

3. Clinician Evaluation via @jwyrwa

   • Create a spreadsheet of the inferred axioms by algorithm and mark them as:
     • Correct/Incorrect
     • Definitely Clinically relevant, Maybe clinically relevant, not clinically relevant

@bill-baumgartner - did I forget anything?

[callahantiff]

@bill-baumgartner - I won't be able to meet during our normal Wednesday time, but can meet anytime today after 1pm and anytime tomorrow (for the most part). Let me know that works best for you!

[bill-baumgartner]

Evaluation steps look good. What was the result of ./owltools -h?
I can meet @ 1pm today. Tomorrow looks dicey due to the weather forecast.

[callahantiff]

Evaluation steps look good. What was the result of ./owltools -h?
I can meet @ 1pm today. Tomorrow looks dicey due to the weather forecast.

I have something at 1pm. I could do 12pm or 2pm. Do either of those work?

The result of running ./owltools -h indicated that of our list of potential ontologies, only ELK and Pellet were included.

[bill-baumgartner]

Noon is great!

[callahantiff]

@bill-baumgartner - following up from our meeting yesterday. Can you please take a look at OWLTools to see how difficult it might be to add a reasoner from the OWLAPI? Thank you!!

[bill-baumgartner]

@callahantiff - can you post the list of reasoners we selected?

[callahantiff]

@callahantiff - can you post the list of reasoners we selected?

No problem, @bill-baumgartner the reasoners we chose are shown in the table below.

Reasoner	Language	OWLTools
ELK	EL	Yes
ELepHant	EL	No
Pellet	DL	Yes
RACER	DL	No
FACT++	DL	No
Chainsaw	DL	No
Konclude	DL	No
Crack	DL	No
TrOWL	DL+EL	No
MORe	DL+EL	No

[bill-baumgartner]

First pass - the biggest issue is that most of the available reasoners support OWL API v3, and not v4 or v5.

• elephant - C code, "Support for OWL API is planned"
• pellet - OWL API v3
• racer - OWL API v3
• chainsaw - OWL API v3
• konclude - no direct connect to OWL API, but may be possible to use: https://www.derivo.de/en/products/konclude/usage-owl-api/
• crack - listed as unmaintained here: http://owl.cs.manchester.ac.uk/tools/list-of-reasoners/
• trowl - OWL API v3
• more - OWL API v3

---

• fact++ - supports OWL API v4
• snorocket - not on our list, but supports OWL API v4

I'll take a look at integrating Konclude first as it looks to be doable, and then perhaps we take a closer look and prioritize the rest.

[callahantiff]

First pass - the biggest issue is that most of the available reasoners support OWL API v3, and not v4 or v5.

• elephant - C code, "Support for OWL API is planned"

• pellet - OWL API v3

• racer - OWL API v3

• chainsaw - OWL API v3

• konclude - no direct connect to OWL API, but may be possible to use: https://www.derivo.de/en/products/konclude/usage-owl-api/

• crack - listed as unmaintained here: http://owl.cs.manchester.ac.uk/tools/list-of-reasoners/

• trowl - OWL API v3

• more - OWL API v3

• fact++ - supports OWL API v4

• snorocket - not on our list, but supports OWL API v4

I'll take a look at integrating Konclude first as it looks to be doable, and then perhaps we take a closer look and prioritize the rest.

Thanks for the update @bill-baumgartner! I think your plan sounds great! Let me know if I can do anything in the meantime! I will also be in tomorrow if you prefer to meet in person.

[callahantiff]

@bill-baumgartner - did you want to touch base on this stuff today?

[bill-baumgartner]

Current status of attempting to use Konclude though the OWL-API is summarized here: ignazio1977/owllink-owlapi#2

Probably not worth meeting today unless you have something you want to discuss.

[bill-baumgartner]

Turns out that Konclude works well from the command-line (in conjunction with Docker). So far I have not figured out how to retrieve explanations for unsatisfiable classes, but it is straightforward to:

• check the consistency of an ontology, e.g.
  docker run --rm --name "konclude" -v /local/path/to/ontologies:/data konclude/konclude consistency -i /data/cl.owl
• classify an ontology, e.g.
  docker run --rm --name "konclude" -v /local/path/to/ontologies:/data konclude/konclude classify -v -i /data/cl.owl -o /data/cl.konclude.out
• and, check the satisfiability of a specific class, e.g.
  docker run --rm --name "konclude" -v /local/path/to/ontologies:/data konclude/konclude satisfiability -v -i /data/cl.owl -x http://purl.obolibrary.org/obo/CL_2000096

[callahantiff]

Turns out that Konclude works well from the command-line (in conjunction with Docker). So far I have not figured out how to retrieve explanations for unsatisfiable classes, but it is straightforward to:

• check the consistency of an ontology, e.g.
  docker run --rm --name "konclude" -v /local/path/to/ontologies:/data konclude/konclude consistency -i /data/cl.owl
• classify an ontology, e.g.
  docker run --rm --name "konclude" -v /local/path/to/ontologies:/data konclude/konclude classify -v -i /data/cl.owl -o /data/cl.konclude.out
• and, check the satisfiability of a specific class, e.g.
  docker run --rm --name "konclude" -v /local/path/to/ontologies:/data konclude/konclude satisfiability -v -i /data/cl.owl -x http://purl.obolibrary.org/obo/CL_2000096

Awesome, thanks Bill! Once I have the graph built I will give this a try! Happy to help dig around to find the output for unsatisfiable classes too.

[callahantiff]

Morning @bill-baumgartner!

Below is a link to download a zipped directory of the ontologies we can use for the reasoner challenge. I have spent the last few days extensively cleaning these files (i.e. removed punning, fixing erroneous Literal typing, correcting inconsistent imported ontology identifiers, etc.). I also went ahead and ran each cleaned file through ELK to make sure that the cleaning didn't inadvertently cause other errors. All of them passed.

Download Link: reasoner_challenge.zip

Directory Contents
Aside from the clean single ontology files, you will also find:

• Each individual, cleaned ontology (n=10)
• A file for each pairwise ontology merge (n=45; naming scheme: ont1_ont2_merged.owl)
• A file of all of the single ontologies merged together (all_ontologies_merged.owl)

Note. All ontologies in the directory contain all original entities (i.e. not had all annotation assertions removed to create the minimal symbolic representation). I figured we should start with them intact and if need be, apply the reduction transformation later on (it's super easy to do).

Question: @bill-baumgartner - do we want all triple-, quad-, ...n-1-wise merge files? It is easy to generate these, but thought I would ask just in case you had something else in mind.

Ok, that's all for now, see you in a few hours! 😄 🎉

[callahantiff]

@bill-baumgartner - here is the Jupyter Notebook I mentioned during our meeting today (incase you are curious).

[callahantiff]

@bill-baumgartner - just wanted to check-in and see how the great reasoner challenge of 2020 was going? Still OK to meet at our usual time tomorrow?

[bill-baumgartner]

Yep, let's meet. I should have some preliminary data to show you.

[callahantiff]

Yep, let's meet. I should have some preliminary data to show you.

@bill-baumgartner - Woo hoo! Sounds great, I am looking forward to it!