Rubric for Assessing XAI Explanation

A survey to test the appropriateness of a rubric in assessing the quality of an XAI explanation

Thank you for agreeing to be part of the 2019-20 Explainable Artificial Intelligence Delphi Group.

I will aggregate everybody’s responses and share the results, along with follow-up questions, for the next stage of the Delphi methodology.

If you can, please respond by 1 March 2020 so that I can include your contribution.

The Plain Language Information Statement is available here. More information on the survey method can be found here.

There are 11 questions in this survey.

Rubric Questions

Case Study

A bank customer requests the reasoning behind her bank loan rejection in order to understand why her loan has been rejected by an AI. 

(We may assume that the customer had no specialized knowledge of loan application processing nor of computer routines ("algorithms") and A.I.)

The explanation he is given was itself provided by an algorithm. This produced both a graphical and a textual explanation for the A.I.'s decision to reject his application.

The Explanation Proferred by the XAI

[Text quoted from the preamble to the XAI's explanation is presented here in blue.]

FICO, a credit scoring company, released an anonymized dataset of Home Equity Line of Credit (HELOC) applications made by real homeowners. A HELOC is a line of credit typically offered by a bank as a percentage of home equity (the difference between the current market value of a home and the outstanding balance of all liens, e.g., mortgages). The customers in this dataset have requested a credit line in the range of $5,000 - $150,000. The fundamental task is to use the information about the applicant in their credit report to predict whether they will make timely payments over a two-year period. This is the machine learning task that we focus on. The machine learning prediction is then used by loan officers to decide whether the homeowner qualifies for a line of credit and, if so, how much credit should be extended. 

Explanation to be Evaluated

Instructions for completing this survey

Please check the most appropriate option for each of the questions below. If you select "unable to judge"  explain why.

1. Is the explanation clearly written with correct spelling and grammar? *

Choose one of the following answers
Please choose only one of the following:

  • The explanation is written clearly with impeccable grammar and spelling
  • The explanation is written clearly with acceptable grammar and spelling
  • The explanation is understandable, however, there are many grammatical errors and spelling mistakes
  • The explanation is not clearly written and it is not understandable
  • Unable to judge (use the comment box to explain why)
Make a comment on your choice here:

2. Does the explanation contain definitions, glossaries, and keys where these are needed? *

Choose one of the following answers
Please choose only one of the following:

  • All unusual and uncommon words are defined, a glossary is provided, and images have a key if they need one
  • A glossary is provided and images have a key
  • There are missing definitions and the key is missing some symbols
  • No definitions, glossaries, or keys are provided even though the explanation is unclear without them
  • Unable to judge (use the comment box to explain why)
Make a comment on your choice here:

3. Is the supporting information well presented, labelled, and understandable? *

Choose one of the following answers
Please choose only one of the following:

  • The information supplied to support the explanation is well presented, labelled, and understandable
  • The information supplied to support the explanation is labelled and understandable
  • The information supplied is mostly labelled and mostly understandable
  • The information supplied to support the explanation is poorly presented and not understandable.
  • Unable to judge (use the comment box to explain why)
Make a comment on your choice here:

4. Can the explanation be acted upon? Was the explanation given in a timely fashion? *

Choose one of the following answers
Please choose only one of the following:

  • The explanation was provided as soon as it was possible and it is clear how to act upon it
  • The explanation is provided in a timely manner and can be acted upon
  • The explanation is provided in a timely manner but it is not clear how to act upon it
  • The explanation is provided late or can not be acted upon
  • Unable to judge (use the comment box to explain why)
Make a comment on your choice here:

5. Are references and citations provided? *

Choose one of the following answers
Please choose only one of the following:

  • All claims are referenced or have citations provided
  • Most claims are referenced or have citations provided
  • Some claims are referenced or have citations provided
  • No references or citations are provided
  • Unable to judge (use the comment box to explain why)
Make a comment on your choice here:

6. Does the explanation refer to facts surrounding the matter being explained? *

Choose one of the following answers
Please choose only one of the following:

  • Frequent references to facts surrounding the matter being explained
  • Some references to facts surrounding the matter being explained
  • Few references to facts surrounding the matter being explained
  • No reference to facts surrounding the matter being explained
  • Unable to judge (use the comment box to explain why)
Make a comment on your choice here:

7. Is the question (or implied question) answered? *

Choose one of the following answers
Please choose only one of the following:

  • The explanation answers the question (or implied question)
  • The explanation mostly answers the question (or implied question)
  • The explanation only partly answers the question (or implied question)
  • The explanation provided is not relevant to the discussion
  • Unable to judge (use the comment box to explain why)
Make a comment on your choice here:

8. Can the logic of the explanation be followed without difficulty? *

Choose one of the following answers
Please choose only one of the following:

  • The logic of the explanation is laid out clearly and it is possible to follow the logic
  • It is possible to follow the logic of the explanation
  • It is possible to follow some of the logic of the explanation, however, some of the explanation is not linked to other parts
  • It is impossible to follow the logic of this explanation
  • Unable to judge (use the comment box to explain why)
Make a comment on your choice here:

9. Is the explanation convincing? *

Choose one of the following answers
Please choose only one of the following:

  • The explanation is convincing
  • The explanation is mostly convincing
  • The explanation is mostly NOT convincing
  • The explanation is NOT convincing
  • Unable to judge (use the comment box to explain why)
Make a comment on your choice here:

10. Have you any further comments about the survey as a whole or about this sample explanation?

Please write your answer here:

Thank you for responding to the survey.

I will aggregate everybody’s responses and share the results, along with follow-up questions, for the next stage of the Delphi methodology.

The Plain Language Information Statement is available here. More information on the survey method can be found here.


03.07.2020 – 00:27

Submit your survey.
Thank you for completing this survey.