T15:

Effective Pedagogical Agent Design

Amy Baylor, Florida State Univ., USA

Brief description:

The purpose of this tutorial is to introduce computer scientist researchers and developers to cognitive science empirical research findings regarding appropriate features for designing effective pedagogical agents in learning environments. The focus will be on reviewing research (particularly controlled experimental studies) and in presenting systematic methods for evaluating pedagogical agents.

Introduction

This session will synthesize research from different areas of cognitive science, including instructional technology research and HCI research as related to pedagogical agents.
The intended outcomes of the tutorial for participants are as follows: 1) to determine key issues for designing effective pedagogical agents after a review of empirical research; and 2) to determine appropriate research methods for evaluating specific pedagogical agents.
What is a pedagogical agent? (brief discussion)
What is the potential value of pedagogical agents? (brief discussion)

PART ONE -- Effective pedagogical agent features

For the first portion of the tutorial (approx 2 hours), the key areas that will be discussed include the following,: a) defining the instructional role of agent; b) setting appropriate media features; c) creating a viable persona; d) determining type and amount of feedback; and, e) implementing multiple pedagogical agents. Throughout these topics, mini-demonstrations will be used to highlight good versus poor pedagogical agent design.

A. The role of agent in learning process

*What is the desired instructional outcome? (e.g., recall, transfer, application)

Why is this a key consideration for pedagogical agent design?

Instructional role of agent

Motivational versus expert

Present information or provide support?

Agent as learning companion or collaborator (e.g., Hietala; Kim)
Agent as mentor (see Baylor, 2000 for overview)

How much “intelligence” is desirable? (e.g., Solomon & Perkins)

Constructivist vs. instructivist planning considerations (e.g., Wasson)

e.g. agent as a cognitive tools or as a scaffold versus “direct teaching”

B. Media Features (e.g., Mayer, 2001, Multimedia Learning)

Voice

Importance of voice as primary medium to convey agent persona
Issue of real human voice versus computer-generated

Image

Gender, ethnicity
Realistic versus cartoon

Animation

Deitic gestures versus affective animations
Role of extraneous animations
Potentially negative effect on cognitive load
e.g., Johnson, W. L., Rickel, J. W., & Lester, J. C. (2000). Animated pedagogical agents: face-to-face interaction in interactive learning environments. International Journal of Artificial Intelligence in Education, 11, 47-78.

C. Key persona features (see Baylor & Ryu, in press for overview)

Emotion, personality

e.g., Kort & Picard

Human-likeness
Credibility
Control (see Baylor, 2001)

D. Feedback issues with pedagogical agents

Desired amount of interaction
Timing of agent feedback
Jeong (2002)’s interaction analysis methodology
Cognitive load issues

E. Multiple agents (see Baylor, 2002a; Baylor, 2002b for overview)

Selection of teachers

e.g., instructivist versus constructivist in MIMIC (Baylor, 2002b)
e.g., different pedagogies in ETOILE (Dillenbourg, P., Mendelsohn, P., & Schneider, D. (1994).
e.g., SCI-WISE [White, B. Y., Shimoda, T. A., & Frederiksen, J. R. (2000). Facilitating students' inquiry learning and metacognitive development through modifiable software advisers. In S. P. Lajoie (Ed.), Computers as Cognitive Tools Vol 2: No More Walls (pp. 97-132). Mahwah, NJ: Lawrence Earlbaum.]

Using agents to represent multiple perspectives on a topic

To promote cognitive conflict and cognitive flexibility
e.g., applying Sharon Derry’s work with video case studies

Cognitive load issues with multiple agents (see Baylor & Chang, 2003

PART TWO -- Best ways to evaluate pedagogical agents

For the second portion of the tutorial (approx 2 hours), the key areas that will be discussed include the following: a) determining the appropriate research method; b) defining the related learning outcomes; c) reviewing existing instrumentation; and, d) designing custom-made instruments. Throughout, there will be case study practice exercises for participants to brainstorm appropriate research methodologies and instrumentation.

A. Appropriateness of research method: Experimental, case studies, usability studies, or other qualitative approaches

Pros/ cons of each depending on constraints

B. Importance of defining learning-related outcome measures

C. Pedagogical agent instruments currently available

Adcock & Van Eck, in press (to assess agent persona features)
Baylor & Ryu, 2003 (to assess perceived agent value for learning and motivation)

D. Designing your own instrument for evaluation

*What are the key constructs for evaluating pedagogical agents?

Performance (e.g., recall, application, transfer)
Attitudes (e.g. disposition, self-efficacy)
Agent’s efficacy at promoting reflection and metacognition
Learner’s perceptions of agents’ value

How are they best measured?
Pros/cons of Likert-scale items versus system-collected data
Other micro-genetic data to collect

Conclusion: Future research questions for pedagogical agents

In conclusion, what we do not know yet about effective features for pedagogical agents will also be briefly discussed.

Target audience:

No prerequisite knowledge is required for this tutorial.

Why of interest to audience:

This topic should be of significant interest to many conference attendees, particularly those that are not familiar with the educational-related instructional technology and cognitive science research regarding pedagogical agents. The focus on designing systematic research methods to evaluate agents (employing both quantitative and qualitative approaches) should also be of interest, given the importance of evaluation in determining agent effectiveness.

Evidence of scholarship in area:

Dr. Baylor has published extensively in this area and is extremely knowledgeable regarding pedagogical agent research. She has also taught advanced seminars in educational research methodology, particularly as applied to advanced learning technologies. Selected publications include the following:

Baylor, A. L., & Ryu, J. (in press). Does the presence of image and animation enhance pedagogical agent persona? Journal of Educational Computing Research.

Baylor, A. L., & Chang, S. (2002). Pedagogical agents as scaffolds: The role of feedback timing, number of agents, and adaptive feedback. Paper presented at the International Conference of the Learning Sciences, Seattle, WA.

Baylor, A. L. (2002a). Agent-based learning environments for investigating teaching and learning. Journal of Educational Computing Research, 26(3), 249-270.

Baylor, A. L. (2002b). Expanding preservice teachers' metacognitive awareness of instructional planning through pedagogical agents. Educational Technology Research & Development, 50(2), 5-22.

Baylor, A. L. (2001). Permutations of control: Cognitive guidelines for agent-based learning environments. Journal of Interactive Learning Research, 12(4), 403-425.

Baylor, A. L. (2000a). Beyond butlers: Intelligent agents as mentors. Journal of Educational Computing Research, 22(4), 373-382.

Baylor, A. L. (2000b). Cognitive strategies for technology and training. TechTrends, 44(5), 13-15.

Baylor, A. L. (1999). Intelligent agents as cognitive tools. Educational Technology, 39(2), 36-40.

Dr. Baylor founded and directs the NSF-sponsored PALS (Pedagogical Agent Learning Systems) research lab at Florida State University. See http://pals.fsu.edu for more information. She has served on expert panels for agents at International Conference of Advanced Learning Technologies (ICALT), on "Agents, Believability and Embodiment in Advanced Learning Environments,” as a keynote speaker at the NSF-sponsored Agent Institute, and as an invited speaker and visiting scholar researching pedagogical agents at the Knowledge Based Systems Institute at the University of Hannover, Germany.