A survey to test the appropriateness of a rubric in assessing the quality of an XAI explanation
Please note: this paragraph will be edited to be more specific to the Amazon Mechanical Turk cohort when ethics approval comes through
Thank you for agreeing to be part of the 2019-20 Explainable Artificial Intelligence Delphi Group.
Please respond by 1 September 2020.
The Plain Language Information Statement is available here.
This survey will ask you to consider two case studies and answer a number of short questions relating to the case studies.
A printable version of the survey is available here
In the survey, I have removed questions that repeat themselves. I did this because I do not wish to waste your time by making you review questions that are not new. I have left placeholder text to indicate where duplicate questions have been removed.
Please note: this paragraph will be edited to be more specific to the Amazon Mechanical Turk cohort when ethics approval comes through
There are 35 questions in this survey.
A customer, Jason, requests an explanation from the bank to clarify the decision by their Artificial Intelligence (A.I.) algorithm to reject his loan to understand why his application for a loan was denied.
For this case study, we assume that Jason had no specialized knowledge of loan application processing, or computer routines ("algorithms"), or A.I.
The explanation Jason was given was itself created by an algorithm. The explanation contains both a graphical and a textual explanation of the A.I.'s decision to reject his application.
Start of Explanation to Evaluate
End of Explanation to Evaluate
Why did Jason receive this explanation?
*What does Jason need to do next to be accepted for the loan next time?
*What was the main reason Jason was rejected?
*What rating (out of 10) would you give the explanation given to Jason?
*0 Not an explanation at all
10 A good and clear explanation
Services Australia (formerly known as Centrelink) is a program of the Australian government that provides social security payments and services to Australians. Centrelink sent out a letter informing people that they needed to explain the difference in the amount of money they declared to Centrelink and the money they reported to the Australian Tax Office (ATO). If the recipient of the letter could not explain the difference, they would get into debt with Centrelink.
This Centrelink compliance letter was personalised using mail merge (more information on mail merge is here).
If the recipient of the letter does not respond to the letter, will they have a debt with Centrelink?
*What does the recipient of the letter need to do next?
*Why did the recipient of the letter receive this letter?
*0 Not an explanation at all
10 A good and clear explanation
A customer, Jason, requests an explanation from the bank to clarify the decision by their Artificial Intelligence (A.I.) algorithm to reject his loan to understand why his application for a loan was denied.
For this case study, we assume that Jason had no specialized knowledge of loan application processing, or computer routines ("algorithms"), or A.I.
The explanation Jason was given was itself created by an algorithm. The explanation contains both a graphical and a textual explanation of the A.I.'s decision to reject his application.
Start of Explanation to Evaluate
End of Explanation to Evaluate
Please check the most appropriate option for each of the questions below. If you need to explain why you chose the option you did, use the comments box.
A good discussion and definition of "supporting detail" can be found here
In this question "acted upon" is defined as:
"To take action that is strongly influenced by certain information or advice."
acted upon. (n.d.) Farlex Dictionary of Idioms. (2015). Retrieved August 5, 2020, from https://idioms.thefreedictionary.com/acted+upon
A reference tells the readers where the information came from and gives the readers details about the source so that they have a good understanding of what kind of source it is and could find the source themselves if necessary.
NC State University. (2004). Citations and references: Documenting your sources. LabWrite for Students. https://labwrite.ncsu.edu/res/res-citsandrefs.html
Are relevant facts mentioned in the explanation?
*
This question is a placeholder for the section of the survey, which asks rubric questions about the Centrelink case study (Case Study 2).
The Centrelink case study presentation should be identical to the one presented in the raw score section.
The rubric section should be identical to the other rubric section.
To prevent duplication, I have removed the duplicates from this section, leaving only the placeholder.
For example
For example
Before completing this survey, how much do you know about Centrelink or RoboDebt?
*
Thank you for responding to the survey.
The Plain Language Information Statement is available here.
Please note: this paragraph will be edited to be more specific to the Amazon Mechanical Turk cohort when ethics approval comes through
16 Is the explanation clear? *