Explainable Artificial Intelligence Survey

A survey to test the appropriateness of a rubric in assessing the quality of an XAI explanation.

Thank you for participating in this Mechanical Turk survey.

This survey will ask you to consider two case studies and answer a number of short questions relating to the case studies.

If you have any questions please contact us at XAIMTurkSurvey@federation.edu.au

Please note: We are based in Australia. We will be monitoring this email between 9 am and 5 pm AEDT. This is equivalent to 6 pm to 1 am in New York EDT (22:00 to 06:00 UTC). 

There are 50 questions in this survey.

Plain Language Information Statement

1

SCHOOL OF ENGINEERING, INFORMATION TECHNOLOGY AND PHYSICAL SCIENCES

 

PROJECT TITLE:

Measuring the Effectiveness of Explanations of the Decisions of Artificial Intelligence (AI) Algorithms

PRINCIPAL RESEARCHER:

Dr Peter Vamplew

OTHER/STUDENT RESEARCHERS:

Dr Cameron Foale

Dr Richard Dazeley

Charlotte Young

 

You have been invited to participate in a survey that we hope will refine a rubric for use in evaluating the explanations produced by an Explainable Artificial Intelligence (XAI) system. You were selected because you agreed to participate in this study. The study was advertised on Amazon Mechanical Turk.

This survey is part of an Explainable Artificial Intelligence Research Project. You will be asked to read two example explanations and answer questions about them and their quality. You will also be asked some questions about yourself to help us to understand the demographics of this survey. This survey should take up to 30 minutes to complete, and you will be compensated USD 4 upon completion.

Involvement in the project is voluntary, and you are free to withdraw their consent to participate at any time and to withdraw any unprocessed data previously supplied. The completion of a survey indicates implied consent to participate. However, withdrawal of participation will lead to you not getting paid through Amazon Mechanical Turk. If consent is withdrawn after data has been aggregated and processed it will not be possible to withdraw non-identifiable data, although consent can still be withdrawn

The emails will include questions about your opinions on marking rubrics and XAI. You will also be required to answer some comprehension questions to make certain you read the provided case studies. You will also be asked some demographic questions.

There will be an opportunity for participants to preview results and transcripts and to withdraw or amend (if appropriate) any data during or at the end of the process or any unprocessed data previously supplied.

Data confidentiality will be maintained by only storing the information on the Federation University Australia email system, and on the Federation University Australia computer network and only researchers named on this project will have access to this data. The data will be stored indefinitely. Collected data will be confidential, and no identifying information will be used in any publication arising from the research without permission. Be aware that in participating in this research, your de-identified data may be used to inform future research.

The results will be made public and published in Charlotte Young’s thesis and journal articles related to the topic.

Disseminated results will not include information that identifies individual participants (unless specific permission has been obtained).

Confidentiality of information offered is subject to legal limitations (e.g., subpoena, freedom of information claim, or mandatory reporting in some professions).

For Federation University Australia participants the project has received clearance by the HREC.

There should not be any discomfort caused by this research. However, there is a slight risk that discussing the topics in these case studies, such as the rejection of a loan application or a request for payment from a social security department, may cause you to become upset or disturbed.

If you feel distressed or worried about such matters, you should seek help from the appropriate local support group. A list of crisis hotlines for many countries is available here: https://www.opencounseling.com/suicide-hotlines

For follow up technical support, please contact us on XAIMTurkSurvey@federation.edu.au. Emails and Turkopticon (an Amazon Mechanical Turk worker website) will be monitored so that researchers can respond quickly to any concerns participants may have.

Please note: We are based in Australia. We will be monitoring this email and Turkopticon between 9 am and 5 pm AEDT. This is equivalent to 6 pm to 1 am in New York EDT (22:00 to 06:00 UTC).

If you have any questions, or you would like further information regarding the project titled  Measuring the Effectiveness of Explanations of the Decisions of Artificial Intelligence (AI) Algorithms, please contact the Principal Researcher, Dr Peter Vamplew of the School of Science, Engineering, Information Technology:

EMAIL: XAIMTurkSurvey@federation.edu.au

PH: +61 3 5327 9616

 

 

Should you (i.e. the participant) have any concerns about the ethical conduct of this research project, please contact the Federation University Coordinator Research Ethics, Research Services, Federation University Australia,

P O Box 663 Mt Helen Vic 3353

Telephone:  (03)  5327 9765

Email: research.ethics@federation.edu.au

 

CRICOS Provider Number 00103D

If you consent to participate in this survey, please select "next" and continue with the survey, otherwise please exit this survey. 

If you consent to participate in this survey, please select "next" and continue with the survey, otherwise please exit this survey. 

Initial Questions on Case Study 1

2

Case Study 1

A customer, Jason, requests an explanation from the bank to clarify the decision by their Artificial Intelligence (A.I.) algorithm to reject his loan to understand why his application for a loan was denied.

For this case study, we assume that Jason had no specialized knowledge of loan application processing, or computer routines ("algorithms"), or A.I.

The explanation Jason was given was itself created by an algorithm. The explanation contains both a graphical and a textual explanation of the A.I.'s decision to reject his application.

The Explanation Offered by the XAI

Start of Explanation to Evaluate

End of Explanation to Evaluate

3 What does the "Consolidated risk markers" value need to be (approximately)? *

Please choose only one of the following:

  • 58
  • 2
  • 72
  • 99
This question is to check that you read Case Study 1. It has only one right answer. Please choose the most correct answer.

4

Why did Jason receive this explanation? *

Please write your answer here:

5

What does Jason need to do next to be accepted for the loan next time? *

Please write your answer here:

6

What was the main reason Jason was rejected? *

Please write your answer here:

7

What rating (out of 10) would you give the explanation given to Jason? *

Please write your answer here:

0 Not an explanation at all

10 A good and clear explanation

Initial Questions on Case Study 2

8

Case Study 2

Services Australia (formerly known as Centrelink) is a program of the Australian government that provides social security payments and services to Australians. Centrelink sent out a letter informing people that they needed to explain the difference in the amount of money they declared to Centrelink and the money they reported to the Australian Tax Office (ATO). If the recipient of the letter could not explain the difference, they would get into debt with Centrelink.

This Centrelink compliance letter was personalised using mail merge (more information on mail merge is here).

Centrelink Letter (opens in a new tab)

9 How much money did the recipient of the letter earn from "Super Sparkle Cleaning"? *

Please choose only one of the following:

  • $200
  • $400
  • $600
  • $800
This question is to check that you read Case Study 2. It has only one right answer. Please choose the most correct answer.

10

If the recipient of the letter does not respond to the letter, will they have a debt with Centrelink? *

Please write your answer here:

11

What does the recipient of the letter need to do next? *

Please write your answer here:

12

Why did the recipient of the letter receive this letter? *

Please write your answer here:

13 What rating (out of 10) would you give the explanation in the letter? *

Please write your answer here:

0 Not an explanation at all

10 A good and clear explanation

Explanation of Rubric Section

14 You will now be asked to rate the two case studies you previously looked at using a different rating system. This new rating system will help us to understand your previous ratings better. 

Rubric Marking of Case Study 1

15

Case Study 1

A customer, Jason, requests an explanation from the bank to clarify the decision by their Artificial Intelligence (A.I.) algorithm to reject his loan to understand why his application for a loan was denied.

For this case study, we assume that Jason had no specialized knowledge of loan application processing, or computer routines ("algorithms"), or A.I.

The explanation Jason was given was itself created by an algorithm. The explanation contains both a graphical and a textual explanation of the A.I.'s decision to reject his application.

The Explanation Offered by the XAI

Start of Explanation to Evaluate

End of Explanation to Evaluate

16

Instructions for completing this part of the survey

Please check the most appropriate option for each of the questions below. If you need to explain why you chose the option you did, use the comments box.

17 Is the explanation clear? *

Please choose only one of the following:

  • The explanation is clear
  • The explanation is mostly clear
  • The explanation is only clear in parts
  • The explanation is not clear
Make a comment on your choice here:

18 Is the explanation written with correct spelling and grammar? *

Please choose only one of the following:

  • The explanation has no grammatical errors and no spelling mistakes
  • The explanation has some grammatical errors and spelling mistakes
  • The explanation has many grammatical errors and spelling mistakes
  • The explanation is not readable because of the grammatical errors and spelling mistakes
Make a comment on your choice here:

19 Does the explanation define the uncommon words and symbols it uses? *

Please choose only one of the following:

  • All unusual and uncommon words and symbols are defined
  • Only some words and symbols are defined
  • No words and symbols are defined, even though the explanation is unclear without them
  • No definitions or keys were needed
Make a comment on your choice here:

20 Is the supporting information (or detail) well presented and understandable? *

Please choose only one of the following:

  • The information supplied to support the explanation is clearly presented and understandable
  • The information supplied is mostly clearly presented and mostly understandable
  • The information supplied to support the explanation is poorly presented and not understandable.
  • There is no supporting information
Make a comment on your choice here:

A good discussion and definition of "supporting detail" can be found here

21 Can the explanation be acted upon? *

Please choose only one of the following:

  • It is clear how to act upon the explanation and there are instructions for the reader on what to next
  • The explanation can be acted upon
  • It is not clear how to act upon the explanation but it can be acted upon
  • The explanation can not be acted upon
Make a comment on your choice here:

In this question "acted upon" is defined as: 
"To take action that is strongly influenced by certain information or advice."

Acted upon. (2020). In TheFreeDictionary.com.
https://idioms.thefreedictionary.com/acted+upon

22 Are references provided? *

Please choose only one of the following:

  • All claims that need it are referenced
  • Some claims that need it are referenced
  • No references are provided even though there are claims that need references
  • No references are needed to support this explanation, and so no references are provided
Make a comment on your choice here:

reference tells the readers where the information came from and gives the readers details about the source so that they have a good understanding of what kind of source it is and could find the source themselves if necessary.

NC State University. (2004). Citations and references: Documenting your sources. LabWrite for Students. https://labwrite.ncsu.edu/res/res-citsandrefs.html

23

Are relevant facts mentioned in the explanation?

*

Please choose only one of the following:

  • There are frequent references to relevant facts
  • There are some references to relevant facts
  • There are few references to relevant facts
  • There are no reference to relevant facts
Make a comment on your choice here:

Relevant facts are facts that add to the discussion, and that are connected to the explanation.

 

24 Is the question (or implied question) answered? *

Please choose only one of the following:

  • The explanation answers the question (or implied question)
  • The explanation mostly answers the question (or implied question)
  • The explanation only partly answers the question (or implied question)
  • The explanation provided is not relevant to the discussion
Make a comment on your choice here:
Does the explanation address what the audience to want to know?

25 Can the explanation's reasoning be followed without difficulty? *

Please choose only one of the following:

  • The reasoning of the explanation is laid out clearly and it is possible to follow the argument
  • It is possible to follow the reasoning of the explanation
  • It is possible to follow some of the reasoning of the explanation, however, some of the explanation is not linked to other parts
  • It is impossible to follow the reasoning of this explanation
Make a comment on your choice here:

26 Is the explanation convincing? *

Please choose only one of the following:

  • The explanation is convincing
  • The explanation is mostly convincing
  • The explanation is mostly NOT convincing
  • The explanation is NOT convincing
Make a comment on your choice here:
Are you, the survey taker, convinced by this explanation?

27 Have you any further comments about the survey as a whole or this sample explanation?

Please write your answer here:

 

 

Rubric Marking of Case Study 2

28

Case Study 2

Services Australia (formerly known as Centrelink) is a program of the Australian government that provides social security payments and services to Australians. Centrelink sent out a letter informing people that they needed to explain the difference in the amount of money they declared to Centrelink and the money they reported to the Australian Tax Office (ATO). If the recipient of the letter could not explain the difference, they would get into debt with Centrelink.

This Centrelink compliance letter was personalised using mail merge (more information on mail merge is here).

Centrelink Letter (opens in a new tab)

29

Instructions for completing this part of the survey

Please check the most appropriate option for each of the questions below. If you need to explain why you chose the option you did, use the comments box.

30 Is the explanation clear? *

Please choose only one of the following:

  • The explanation is clear
  • The explanation is mostly clear
  • The explanation is only clear in parts
  • The explanation is not clear
Make a comment on your choice here:

31 Is the explanation written with correct spelling and grammar? *

Please choose only one of the following:

  • The explanation has no grammatical errors and no spelling mistakes
  • The explanation has some grammatical errors and spelling mistakes
  • The explanation has many grammatical errors and spelling mistakes
  • The explanation is not readable because of the grammatical errors and spelling mistakes
Make a comment on your choice here:

32 Does the explanation define the uncommon words and symbols it uses? *

Please choose only one of the following:

  • All unusual and uncommon words and symbols are defined
  • Only some words and symbols are defined
  • No words and symbols are defined, even though the explanation is unclear without them
  • No definitions or keys were needed
Make a comment on your choice here:

33 Is the supporting information (or detail) well presented and understandable? *

Please choose only one of the following:

  • The information supplied to support the explanation is clearly presented and understandable
  • The information supplied is mostly clearly presented and mostly understandable
  • The information supplied to support the explanation is poorly presented and not understandable.
  • There is no supporting information
Make a comment on your choice here:

A good discussion and definition of "supporting detail" can be found here

34 Can the explanation be acted upon? *

Please choose only one of the following:

  • It is clear how to act upon the explanation and there are instructions for the reader on what to next
  • The explanation can be acted upon
  • It is not clear how to act upon the explanation but it can be acted upon
  • The explanation can not be acted upon
Make a comment on your choice here:

In this question "acted upon" is defined as: 
"To take action that is strongly influenced by certain information or advice."

Acted upon. (2020). In TheFreeDictionary.com.
https://idioms.thefreedictionary.com/acted+upon

 

35 Are references provided? *

Please choose only one of the following:

  • All claims that need it are referenced
  • Some claims that need it are referenced
  • No references are provided even though there are claims that need references
  • No references are needed to support this explanation, and so no references are provided
Make a comment on your choice here:

reference tells the readers where the information came from and gives the readers details about the source so that they have a good understanding of what kind of source it is and could find the source themselves if necessary.

NC State University. (2004). Citations and references: Documenting your sources. LabWrite for Students. https://labwrite.ncsu.edu/res/res-citsandrefs.html

36

Are relevant facts mentioned in the explanation?

*

Please choose only one of the following:

  • There are frequent references to relevant facts
  • There are some references to relevant facts
  • There are few references to relevant facts
  • There are no reference to relevant facts
Make a comment on your choice here:
Relevant facts are facts that add to the discussion, and that are connected to the explanation.  
 

37 Is the question (or implied question) answered? *

Please choose only one of the following:

  • The explanation answers the question (or implied question)
  • The explanation mostly answers the question (or implied question)
  • The explanation only partly answers the question (or implied question)
  • The explanation provided is not relevant to the discussion
Make a comment on your choice here:
Does the explanation address what the audience to want to know?

38 Can the explanation's reasoning be followed without difficulty? *

Please choose only one of the following:

  • The reasoning of the explanation is laid out clearly and it is possible to follow the argument
  • It is possible to follow the reasoning of the explanation
  • It is possible to follow some of the reasoning of the explanation, however, some of the explanation is not linked to other parts
  • It is impossible to follow the reasoning of this explanation
Make a comment on your choice here:

39 Is the explanation convincing? *

Please choose only one of the following:

  • The explanation is convincing
  • The explanation is mostly convincing
  • The explanation is mostly NOT convincing
  • The explanation is NOT convincing
Make a comment on your choice here:
Are you, the survey taker, convinced by this explanation?

40 Have you any further comments about the survey as a whole or this sample explanation?

Please write your answer here:

 

 

Census of participants

41 What is your age? *

Please write your answer here:

If you prefer not to say, please enter 0

42 What country do you live in? *

Please write your answer here:

If you prefer not to say, please enter "private"

43 What is your occupation? *

Please write your answer here:

If you do not wish to state your occupation please enter "Private"

44 How often do you consume entertainment about AI? *

Please choose only one of the following:

  • I have never consumed entertainment that features AI
  • I seldom consume entertainment that features AI
  • I sometimes consume entertainment that features AI
  • I often consume entertainment that features AI
Make a comment on your choice here:

For example

  • Television shows
  • Movies
  • Fiction books
  • Games
  • Comics

45 How often do you consume non-fiction about AI? *

Please choose only one of the following:

  • I have never consumed non-fiction about AI
  • I seldom consume non-fiction about AI
  • I sometimes consume non-fiction about AI
  • I often consume non-fiction about AI
Make a comment on your choice here:

For example

  • Journal articles
  • Newspaper articles
  • Lectures

46 How much do you know about loan application processes? *

Please choose only one of the following:

  • I have not applied for a loan and I have not seen other people apply for loans
  • I have watched other people apply for loans
  • I have applied for a loan myself
  • Prefer not to say
Make a comment on your choice here:

47

Before completing this survey, how much do you know about Centrelink or RoboDebt? *

Please choose only one of the following:

  • I know nothing about Centrelink Australia or RoboDebt
  • I have seen some discussion online about Centrelink Australia or RoboDebt
  • I have read a news article about Centrelink Australia or RoboDebt
  • I have experienced RoboDebt or I know someone who has experienced RoboDebt
  • I have worked in Centrelink
  • Prefer not to say
Make a comment on your choice here:

Further Comments

48 Do you have any more comments to make?

Please write your answer here:

Validation

49 Your unique code is {sum(INSERTANS:828584X171X1296,INSERTANS:828584X172X1308,38)*887} *

If you have any difficulties with this code, please email us at XAIMTurkSurvey@federation.edu.au

Please note: We are based in Australia. We will be monitoring this email and Turkopticon between 9 am and 5 pm AEDT. This is equivalent to 6 pm to 1 am in New York EDT (22:00 to 06:00 UTC)

50

The code above will need to be entered on the HIT on the Amazon Mechanical Turk website to confirm you completed this survey.

You should write down the access code.

Please check yes to confirm you have gotten your access code.

*

Please choose only one of the following:

  • Yes
  • No
Please check yes to confirm you have gotten your access code.

Thank you for responding to the survey.

If you have any questions please contact us at XAIMTurkSurvey@federation.edu.au

Please note: We are based in Australia. We will be monitoring this email between 9 am and 5 pm AEDT. This is equivalent to 6 pm to 1 am in New York EDT (22:00 to 06:00 UTC).


05.05.2021 – 10:27

Submit your survey.
Thank you for completing this survey.