A survey to assess the quality of an XAI explanation.
Thank you for participating in this Mechanical Turk survey.
This survey will ask you to consider one case study and answer a number of short questions relating to the case study.
If you have any questions please contact us at XAIMTurkSurvey@federation.edu.au
Please note: We are based in Australia. We will be monitoring this email between 9 am and 5 pm AEST. This is equivalent to 7 pm to 3 am in New York EDT (23:00 to 07:00 UTC).
There are 30 questions in this survey.
SCHOOL OF ENGINEERING, INFORMATION TECHNOLOGY AND PHYSICAL SCIENCES
PROJECT TITLE: |
Measuring the Effectiveness of Explanations of the Decisions of Artificial Intelligence (AI) Algorithms |
PRINCIPAL RESEARCHER: |
Dr Peter Vamplew |
OTHER/STUDENT RESEARCHERS: |
Dr Cameron Foale Dr Richard Dazeley Dr Francisco Cruz Charlotte Young |
You have been invited to participate in a survey that we hope will refine a rubric for use in evaluating the explanations produced by an Explainable Artificial Intelligence (XAI) system. You were selected because you agreed to participate in this study. The study was advertised on Amazon Mechanical Turk.
This survey is part of an Explainable Artificial Intelligence Research Project. You will be asked to read one example explanation and answer questions about it and its quality. Some of these questions will evaluate if you have read the example explanation, if you fail these questions you may be judged to be not paying sufficient attention for your answers to count, and therefore you could possibly not get paid.
You will also be asked some questions about yourself to help us to understand the demographics of this survey.
This survey should take up to 15 minutes to complete, and you will be compensated USD 2 upon completion.
Involvement in the project is voluntary, and you are free to withdraw their consent to participate at any time and to withdraw any unprocessed data previously supplied. The completion of a survey indicates implied consent to participate. However, withdrawal of participation will lead to you not getting paid through Amazon Mechanical Turk. If consent is withdrawn after data has been aggregated and processed it will not be possible to withdraw non-identifiable data, although consent can still be withdrawn.
There will be an opportunity for participants to preview results and transcripts and to withdraw or amend (if appropriate) any data during or at the end of the process or any unprocessed data previously supplied.
Data confidentiality will be maintained by only storing the information on the Federation University Australia system, and on the Federation University Australia computer network and only researchers named on this project will have access to this data. The data will be stored indefinitely. Collected data will be confidential, and no identifying information will be used in any publication arising from the research without permission. Be aware that in participating in this research, your de-identified data may be used to inform future research.
The results will be made public and published in Charlotte Young’s thesis and journal articles related to the topic.
Disseminated results will not include information that identifies individual participants (unless specific permission has been obtained).
Confidentiality of information offered is subject to legal limitations (e.g., subpoena, freedom of information claim, or mandatory reporting in some professions).
For Federation University Australia participants the project has received clearance by the HREC.
There should not be any discomfort caused by this research. However, there is a slight risk that discussing the topics in these case studies may cause you to become upset or disturbed.
If you feel distressed or worried about such matters, you should seek help from the appropriate local support group. A list of crisis hotlines for many countries is available here: https://www.opencounseling.com/suicide-hotlines
For follow up technical support, please contact us on XAIMTurkSurvey@federation.edu.au. Emails and Turkopticon (an Amazon Mechanical Turk worker website) will be monitored so that researchers can respond quickly to any concerns participants may have.
Please note: We are based in Australia. We will be monitoring this email and Turkopticon between 9 am and 5 pm AEDT. This is equivalent to 6 pm to 1 am in New York EDT (22:00 to 06:00 UTC).
If you have any questions, or you would like further information regarding the project titled Measuring the Effectiveness of Explanations of the Decisions of Artificial Intelligence (AI) Algorithms, please contact the Principal Researcher, Dr Peter Vamplew of the School of Science, Engineering, Information Technology: EMAIL: XAIMTurkSurvey@federation.edu.au PH: +61 3 5327 9616
|
Should you (i.e. the participant) have any concerns about the ethical conduct of this research project, please contact the Federation University Coordinator Research Ethics, Research Services, Federation University Australia, P O Box 663 Mt Helen Vic 3353 Telephone: (03) 5327 9765 Email: research.ethics@federation.edu.au
CRICOS Provider Number 00103D |
You will be presented with a scenario based on a hide-and-seek game. In this scenario, the drone and the human wearing the green t-shirt are a team, namely, the aerial and ground team-member respectively. Both team-members work collaboratively aiming to target the opponent team, i.e., the human wearing the red t-shirt. The scenario presents 2 figures depicting an initial situation, for instance at time-step t, and a final situation, for instance at time-step t+1, although it might be a future time-step as well.
The figures show the general aerial view of the situation, the drone view camera (see pink boxes for the drone and the drone view) and a prediction of the team-mate and/or enemy behaviour (see red arrows for the predicted movement). The drone can only see small areas and must determine, based on what the human is doing, the best location for it to search. It must provide details of where it is going, why, and what it sees as a dialog to the human team members. It can also recommend an action for the human team member to perform.
Please note: The drone is controlled by an Artificial Intelligence algorithm not a human
Start of Explanation to Evaluate
End of Explanation to Evaluate
What colour shirt is the opponent team wearing? *
What colour shirt is the team with the drone wearing? *
What game are the two humans and the drone playing? *
What rating (out of 10) would you give the explanation? *
0 Not an explanation at all
10 A good and clear explanation
You will be presented with a scenario based on a hide-and-seek game. In this scenario, the drone and the human wearing the green t-shirt are a team, namely, the aerial and ground team-member respectively. Both team-members work collaboratively aiming to target the opponent team, i.e., the human wearing the red t-shirt. The scenario presents 2 figures depicting an initial situation, for instance at time-step t, and a final situation, for instance at time-step t+1, although it might be a future time-step as well.
The figures show the general aerial view of the situation, the drone view camera (see pink boxes for the drone and the drone view) and a prediction of the team-mate and/or enemy behaviour (see red arrows for the predicted movement). The drone can only see small areas and must determine, based on what the human is doing, the best location for it to search. It must provide details of where it is going, why, and what it sees as a dialog to the human team members. It can also recommend an action for the human team member to perform.
Please note: The drone is controlled by an Artificial Intelligence algorithm not a human
Start of Explanation to Evaluate
End of Explanation to Evaluate
Please check the most appropriate option for each of the questions below. If you need to explain why you chose the option you did, use the comments box.
A good discussion and definition of "supporting detail" can be found here
In this question "acted upon" is defined as:
"To take action that is strongly influenced by certain information or advice."
Acted upon. (2020). In TheFreeDictionary.com.
https://idioms.thefreedictionary.com/acted+upon
A reference tells the readers where the information came from and gives the readers details about the source so that they have a good understanding of what kind of source it is and could find the source themselves if necessary.
NC State University. (2004). Citations and references: Documenting your sources. LabWrite for Students. https://labwrite.ncsu.edu/res/res-citsandrefs.html
Are relevant facts mentioned in the explanation? *
Relevant facts are facts that add to the discussion, and that are connected to the explanation.
For example
For example
If you have any difficulties with this code, please email us at XAIMTurkSurvey@federation.edu.au
Please note: We are based in Australia. We will be monitoring this email and Turkopticon between 9 am and 5 pm AEDT. This is equivalent to 6 pm to 1 am in New York EDT (22:00 to 06:00 UTC)
The code above will need to be entered on the HIT on the Amazon Mechanical Turk website to confirm you completed this survey.
You should write down the access code.
Please check yes to confirm you have gotten your access code. *
Thank you for responding to the survey.
If you have any questions please contact us at XAIMTurkSurvey@federation.edu.au
Please note: We are based in Australia. We will be monitoring this email between 9 am and 5 pm AEST. This is equivalent to 7 pm to 3 am in New York EDT (23:00 to 07:00 UTC).
11 Is the explanation clear? *