T15:

EffectivePedagogical Agent Design

Amy Baylor, Florida State Univ., USA

Briefdescription:

The purpose of this tutorial is to introducecomputer scientist researchers and developers to cognitive science empiricalresearch findings regarding appropriate features for designing effectivepedagogical agents in learning environments. Thefocus will be on reviewing research (particularly controlled experimentalstudies) and in presenting systematic methods for evaluating pedagogical agents.

Introduction

This session will synthesize research from different areas of cognitive science, including instructional technology research and HCI research as related to pedagogical agents.
The intended outcomes of the tutorial for participants are as follows: 1) to determine key issues for designing effective pedagogical agents after a review of empirical research; and 2) to determine appropriate research methods for evaluating specific pedagogical agents.
What is a pedagogical agent? (brief discussion)
What is the potential value of pedagogical agents? (brief discussion)

PARTONE -- Effective pedagogical agentfeatures

For the first portion of thetutorial (approx 2 hours), the key areas that will be discussed include thefollowing,: a) defining theinstructional role of agent; b) setting appropriate media features; c)creating a viable persona; d) determining type and amount of feedback; and, e)implementing multiple pedagogical agents. Throughout these topics,mini-demonstrations will be used to highlight good versus poor pedagogical agentdesign.

A. Therole of agent in learning process

*What is the desired instructional outcome? (e.g., recall, transfer, application)

Why is this a key consideration for pedagogical agent design?

Instructional role of agent

Motivational versus expert

Present information or provide support?

Agent as learning companion or collaborator (e.g., Hietala; Kim)
Agent as mentor (see Baylor, 2000 for overview)

How much “intelligence” is desirable? (e.g., Solomon & Perkins)

Constructivist vs. instructivist planning considerations (e.g., Wasson)

e.g. agent as a cognitive tools or as a scaffold versus “direct teaching”

B. Media Features (e.g., Mayer, 2001, MultimediaLearning)

Voice

Importance of voice as primary medium to convey agent persona
Issue of real human voice versus computer-generated

Image

Gender, ethnicity
Realistic versus cartoon

Animation

Deitic gestures versus affective animations
Role of extraneous animations
Potentially negative effect on cognitive load
e.g., Johnson, W. L., Rickel, J. W., & Lester, J. C. (2000). Animated pedagogical agents: face-to-face interaction in interactive learning environments. International Journal of Artificial Intelligence in Education, 11, 47-78.

C. Keypersona features (see Baylor & Ryu, in press for overview)

Emotion, personality

e.g., Kort & Picard

Human-likeness
Credibility
Control (see Baylor, 2001)

D.Feedback issues with pedagogical agents

Desired amount of interaction
Timing of agent feedback
Jeong (2002)’s interaction analysis methodology
Cognitive load issues

E. Multiple agents (see Baylor, 2002a; Baylor,2002b for overview)

Selection of teachers

e.g., instructivist versus constructivist in MIMIC (Baylor, 2002b)
e.g., different pedagogies in ETOILE (Dillenbourg, P., Mendelsohn, P., & Schneider, D. (1994).
e.g., SCI-WISE [White, B. Y., Shimoda, T. A., & Frederiksen, J. R. (2000). Facilitating students' inquiry learning and metacognitive development through modifiable software advisers. In S. P. Lajoie (Ed.), Computers as Cognitive Tools Vol 2: No More Walls (pp. 97-132). Mahwah, NJ: Lawrence Earlbaum.]

Using agents to represent multiple perspectives on a topic

To promote cognitive conflict and cognitive flexibility
e.g., applying Sharon Derry’s work with video case studies

Cognitive load issues with multiple agents (see Baylor & Chang, 2003

PARTTWO -- Best ways to evaluatepedagogical agents

For the second portion of thetutorial (approx 2 hours), the key areas that will be discussed include thefollowing: a) determining theappropriate research method; b) defining the related learning outcomes; c)reviewing existing instrumentation; and, d) designing custom-made instruments.Throughout, there will be case study practice exercises for participants tobrainstorm appropriate research methodologies and instrumentation.

A. Appropriateness of research method: Experimental, case studies, usability studies, or other qualitativeapproaches

Pros/ cons of each depending on constraints

B. Importance of defining learning-relatedoutcome measures

C. Pedagogical agent instruments currentlyavailable

Adcock & Van Eck, in press (to assess agent persona features)
Baylor & Ryu, 2003 (to assess perceived agent value for learning and motivation)

D. Designing your own instrument for evaluation

*What are the key constructs for evaluating pedagogical agents?

Performance (e.g., recall, application, transfer)
Attitudes (e.g. disposition, self-efficacy)
Agent’s efficacy at promoting reflection and metacognition
Learner’s perceptions of agents’ value

How are they best measured?
Pros/cons of Likert-scale items versus system-collected data
Other micro-genetic data to collect

Conclusion: Future research questions for pedagogical agents

In conclusion, what we do notknow yet about effective features for pedagogical agents will also bebriefly discussed.

Targetaudience:

No prerequisite knowledge isrequired for this tutorial.

Whyof interest to audience:

This topic should be ofsignificant interest to many conference attendees, particularly those that arenot familiar with the educational-related instructional technology and cognitivescience research regarding pedagogical agents. The focus on designing systematicresearch methods to evaluate agents (employing both quantitative and qualitativeapproaches) should also be of interest, given the importance of evaluation indetermining agent effectiveness.

Evidenceof scholarship in area:

Dr.Baylor has published extensively in this area and is extremely knowledgeableregarding pedagogical agent research. Shehas also taught advanced seminars in educational research methodology,particularly as applied to advanced learning technologies. Selected publicationsinclude the following:

Baylor, A.L., & Ryu, J. (in press). Does the presence of image and animation enhancepedagogical agent persona? Journal ofEducational Computing Research.

Baylor, A.L., & Chang, S. (2002). Pedagogicalagents as scaffolds: The role of feedback timing, number of agents, and adaptivefeedback. Paper presented at the International Conference of the LearningSciences, Seattle, WA.

Baylor, A.L. (2002a). Agent-based learning environments for investigating teaching andlearning. Journal of Educational ComputingResearch, 26(3), 249-270.

Baylor, A.L. (2002b). Expanding preservice teachers' metacognitive awareness ofinstructional planning through pedagogical agents. EducationalTechnology Research & Development, 50(2), 5-22.

Baylor, A.L. (2001). Permutations of control: Cognitive guidelines for agent-basedlearning environments. Journal ofInteractive Learning Research, 12(4), 403-425.

Baylor, A. L. (2000a). Beyond butlers: Intelligent agents asmentors. Journal of Educational ComputingResearch, 22(4), 373-382.

Baylor, A.L. (2000b). Cognitive strategies for technology and training. TechTrends,44(5), 13-15.

Baylor, A.L. (1999). Intelligent agents as cognitive tools. EducationalTechnology, 39(2), 36-40.

Dr. Baylor founded and directsthe NSF-sponsored PALS (Pedagogical Agent Learning Systems) research lab atFlorida State University. See http://pals.fsu.edu for more information. She has served on expert panels for agents at International Conference ofAdvanced Learning Technologies (ICALT), on "Agents, Believability andEmbodiment in Advanced Learning Environments,” as a keynote speaker at theNSF-sponsored Agent Institute, and as an invited speaker and visiting scholarresearching pedagogical agents at the Knowledge Based Systems Institute at theUniversity of Hannover, Germany.