Description
Accounting ®rms are intensifying their reliance on experiential learning, and experience increasingly involves the use
of computerized decision aids [Messier, W. (1995) Research in and development of audit decision aids. In R. H. Ashton
& A. H. Ashton, Judgment and decision making in accounting and auditing (pp. 207±230). New York: Cambridge
University Press]. Accountants are expected to learn from automated decision aid use, because the aids are not always
available when dealing with the aid's topical matter, and the knowledge inherent in the aid is needed for competency on
broader issues. To facilitate knowledge acquisition and explain the logic of the underlying processes, computerized
decision aids provide the rationale for their calculations in the form of online explanations. We study how the location
of explanations in a computerized decision aid aects learning from its use. Speci®cally, this research extends the
existing literature by using a framework for the study of learning from decision aid use and by using cognitive load
theory to explain the failure of certain decision aid design alternatives to promote learning.
The e?ects of system design alternatives on the acquisition
of tax knowledge from a computerized tax decision aid
Jacob M. Rose
a,
*, Christopher J. Wolfe
b
a
Department of Accounting, Bryant College, 1150 Douglas Pike, Smith®eld, RI 02917, USA
b
Department of Accounting, Lowry Mays College & Graduate School of Business, Texas A&M University, College Station,
TX 77843-4353, USA
Abstract
Accounting ®rms are intensifying their reliance on experiential learning, and experience increasingly involves the use
of computerized decision aids [Messier, W. (1995) Research in and development of audit decision aids. In R. H. Ashton
& A. H. Ashton, Judgment and decision making in accounting and auditing (pp. 207±230). New York: Cambridge
University Press]. Accountants are expected to learn from automated decision aid use, because the aids are not always
available when dealing with the aid's topical matter, and the knowledge inherent in the aid is needed for competency on
broader issues. To facilitate knowledge acquisition and explain the logic of the underlying processes, computerized
decision aids provide the rationale for their calculations in the form of online explanations. We study how the location
of explanations in a computerized decision aid a?ects learning from its use. Speci®cally, this research extends the
existing literature by using a framework for the study of learning from decision aid use and by using cognitive load
theory to explain the failure of certain decision aid design alternatives to promote learning. We de®ne learning as the
acquisition of problem-type schemata, and an experiment is performed in which cognitive load is manipulated by the
placement of explanations in a computerized tax decision aid to determine its e?ect on schema acquisition. Schemata
are general knowledge structures used for basic comprehension, and cognitive load refers to the burden placed on
working memory when acquiring schemata. We ®nd that increased cognitive load produced by the location of expla-
nations in a decision aid leads to reduced schema acquisition. Our results indicate that when explanations in a com-
puterized decision aid are integrated into its problem solving steps, cognitive load is reduced and users acquire more
knowledge from aid use. This appears to be an important design consideration for accounting ®rms buying or building
computerized decision aids. # 2000 Elsevier Science Ltd. All rights reserved.
This study investigates the determinants of
knowledge acquisition from the use of an auto-
mated tax decision aid. Under the rationale of
eciency and e?ectiveness in decision making,
computer-based decision aids are commonly used
in public accounting (Brown & Eining, 1997;
Messier, 1995). However, assistance in making
decisions is not the only expected function of these
aids. It has been conjectured that experience gar-
nered while using decision aids also promotes
knowledge acquisition, because the aid should
0361-3682/00/$ - see front matter # 2000 Elsevier Science Ltd. All rights reserved.
PI I : S0361- 3682( 99) 00048- 3
Accounting, Organizations and Society 25 (2000) 285±306
www.elsevier.com/locate/aos
* Corresponding author.
E-mail address: [email protected] (J.M. Rose), cjwolfe@
tamu.edu (C.J. Wolfe).
provide an illustration of proper problem solving
method, explanation of the method, and outcome
feedback (Ashton & Willingham, 1988; Pei, Stein-
bart & Reneau, 1994). Pragmatically, experiential
learning from the use of an automated decision
aid is important for at least two reasons: (1) The
aids will not always be conveniently available, and
accounting practitioners often deal with client
scenarios on an ad hoc basis; and (2) The base
knowledge inherent in decision aids must be part
of an accounting professional's repertoire, because
as sta? rise to managerial positions they must be
able to evaluate the output of decision aids in a
broader context.
1
Knowledge has been shown to be a functional
determinant of decision performance (Bonner &
Lewis, 1990; Libby & Tan, 1994). Therefore,
learning from using an automated decision aid is
important to decision performance when making
decisions inside an aid's domain without the use of
the aid and when evaluating the ecacy of an aid's
output. To understand the development of exper-
tise in environments characterized by the use of
automated decision aids, an important implication
is that a detailed understanding of knowledge
acquisition from using such aids is needed ®rst.
Research on learning from computerized deci-
sion aid use has focused on two general questions:
(1) How does experiential learning of computer-
ized decision aid users di?er from hand calculation
groups using traditional text-based materials; and
(2) Can user or system attributes be manipulated
to enhance learning from computerized decision
aid use? Research results on learning di?erences
between computerized decision aid users and hand
calculation groups indicate that hand calculation
treatments outperform aid users when given tra-
ditional text-based materials that facilitate a com-
plete solution to the experimental problems
(Glover, Prawitt & Spilker, 1997; Murphy, 1990).
2
While these ®ndings are compelling, the bene®ts of
decision consistency, eciency, and documenta-
tion apparently outweigh the sub-optimal learning
experience of automated decision aid use, because
accounting ®rms continue to make heavy use of
such aids. Therefore, the more critical question is
the second: Can anything be done to increase
experiential learning when using computerized
decision aids?
Approaching this question from the side of the
decision aid user, the earliest line of research
addressed the possibility that mismatches between
users' knowledge organization and the underlying
structure of the decision aid led to learning de®cits
(Frederick, 1991; Pei & Reneau, 1990; Ricchiute,
1992). These studies found that knowledge acqui-
sition was improved when decision aid structures
matched the knowledge structures of their users.
However, any strategy based on these ®ndings
shifts much of the training burden away from the
experience of using the decision aid.
Modifying the design of a decision aid to
enhance its training capability is superior to train-
ing users on the knowledge structure of a decision
aid, because complete experiential learning
through automated decision aid use is more e-
cient. The design feature inherent in a computer-
ized decision aid to assist learning is the aid's
explanation facility (i.e. a software device that
explains calculation logic). Early research com-
paring the presence or absence of explanations in
a decision aid found explanations inconsequential
to learning (Eining & Dorr, 1991; Murphy, 1990).
Additionally, Steinbart and Accola (1994) found
that more elaborate explanations did not promote
a greater level of learning, and no learning e?ect
was identi®ed for the di?ering placement of
explanations within a decision aid (Mot, 1994;
Odom & Dorr, 1995).
The current study extends the existing literature
by focusing on explanation placement within a
1
Demonstrating ®rm emphasis on experiential learning,
Price Waterhouse Coopers has shifted a signi®cant component
of their tax training to ``structured work assignments in the
oce.'' Deloitte and Touche has also increased its emphasis on
learning through experience. Employee manuals stress that in
today's quickly changing ®nancial environment, on-the-job
training ``is essential to maintain the level of competence
necessary to render excellent service.''
2
While Fedorowicz, Oz and Berger (1992) and Eining and
Dorr (1991) found that computerized decision aid users learned
more than hand calculation groups, equivalency di?erences
existed in the decision support tools for the hand calculation
and computerized treatments.
286 J.M. Rose, C.J. Wolfe / Accounting, Organizations and Society 25 (2000) 285±306
computerized decision aid using theoretical fra-
meworks taken from the educational psychology
and accounting literatures. More precisely, we use
cognitive load theory to explain di?erences in
schema acquisition due to the location of expla-
nations in a computerized decision aid. Schemata
are general knowledge structures used for basic
comprehension and can be de®ned as ``cognitive
constructs that permit problem-solvers to recog-
nize a problem as belonging to a speci®c category
requiring particular moves for completion'' (Tar-
mizi & Sweller, 1988). Cognitive load refers to the
burden placed on working memory when acquir-
ing schemata (Sweller, 1988). In this study, we
conduct an experiment where levels of cognitive
load produced by decision aids are manipulated
by varying the placement of explanations in a
decision aid, and a knowledge acquisition frame-
work is used to investigate the e?ects of cognitive
load on learning from a decision aid.
The framework models the direct and indirect
e?ects of the cognitive load produced by a deci-
sion aid on its user's capacity to learn from the
experience of using the aid, while accounting for
the problem-solving ability of the aid user and the
amount of time spent learning from aid use. A
number of ®ndings emanate from our framework-
based analysis, but the most salient are as follows.
Cognitive load plays an important role when
learning from a computerized decision aid: users
of decision aids producing the least cognitive load
learned the most. We also ®nd that subjects per-
forming hand calculation with text-based instruc-
tions scored 22% higher on our learning
performance measure than did subjects using the
decision aid that produced the lowest cognitive
load, but the hand calculation group spent more
than double the amount of time in the learning
process. When di?erences in time spent learning
and problem-solving ability are statistically con-
trolled, we show that subjects using the decision
aid producing the lowest cognitive load learned
equivalently to subjects performing hand calcula-
tion with text-based instructions.
It appears important for system designers and
accounting ®rms to consider the cognitive load
imposed by a decision aid. Accounting tasks have
speci®c processing requirements, and ®rms have
automated many of these tasks with decision aids.
This research indicates that the placement of
explanations within these aids signi®cantly a?ects
their user's ability to learn from the aid. The
remainder of the paper describes the framework
and hypotheses, followed by a description of the
methods and results. The ®nal section includes
discussion and conclusions, as well as limitations
and suggestions for future research.
1. Framework and Hypotheses
Eq. (1) represents Libby's (1995) model of func-
tional determinants for knowledge:
Knowledge ? f ?Experience; Ability;
Motivation; Environment?
?1?
This research follows that structure de®ning the
constructs and framework as follows.
1.1. Knowledge
Schemata represent the structure and organiza-
tion of knowledge in long-term memory. Psychol-
ogy research indicates that experts have schemata
that allow them to organize and retrieve speci®c
information, while novices lack these schemata.
Similarly, accountants develop detailed schemata
for recognizing and solving accounting problems.
Weber (1980) validated the existence of accounting
schemata in a free recall task requiring auditors to
recall information technology (IT) controls. He
found that experienced IT auditors had higher
levels of cue clustering than students, suggesting
that experienced auditors possessed schemata
which students lacked. Libby (1985) discovered
that possible errors identi®ed by experienced
auditors during the analytical review process
occurred more frequently in practice than the
errors identi®ed by novices. Experts had ®nancial
statement error schemata that allowed them to
recognize reasonable errors. Finally, Frederick
(1991) demonstrated that experienced auditors
organize internal control knowledge based on
transaction ¯ows while novices organize by internal
J.M. Rose, C.J. Wolfe / Accounting, Organizations and Society 25 (2000) 285±306 287
control objectives. The experienced auditors had
schemata that organized their knowledge into a
structure useful for problem solving in the audit-
ing environment. These accounting studies have
generally assumed the existence of schemata, but
have not examined how and under what circum-
stances they are acquired.
In this study, learning is characterized as the
acquisition of schemata. We examine problem-
type schemata which can be de®ned as ``a cogni-
tive construct that permits problem-solvers to
recognize a problem as belonging to a speci®c
category requiring particular moves for comple-
tion'' (Tarmizi & Sweller, 1988).
1.2. Experience
Experience is the treatment in this study and is
de®ned as the quality of an encounter with a
computerized decision aid. The encounter pro-
vides both ®rst-hand and second-hand experiential
learning opportunities. Completion of a problem
using the decision aid represents a ®rst hand
encounter, and reading the explanation o?ered by
the decision aid can be construed as a second-
hand encounter, if it educates the decision aid user
in a broader context. In this research, we manip-
ulate the quality of experience by manipulating the
location of a decision aid's explanations.
Changing the location of explanations in a
decision aid should induce cognitive load and
a?ect schema acquisition. Cognitive load stems
from the interrelationship between working mem-
ory and long-term memory. When acquiring sche-
mata, working memory is used to actively process
new information before it is stored in long-term
memory. Therefore, working memory de®nes the
limits of schema acquisition as the processing
bottleneck (Mousavi, Low & Sweller, 1995; Swel-
ler, 1988, 1993). In experiential learning, problem
solving space in working memory crowds out that
needed for schema acquisition. This burden on
working memory is referred to as cognitive load.
We manipulate the level of cognitive load
through split attention e?ects. Attention is split
when problem solvers must hold information from
one source in working memory while attempting
to evaluate information from one or more other
sources. Splitting attention reduces the memory
available for acquiring schemata, because it
requires information to be held in working mem-
ory and then combined with other information
(Tarmizi & Sweller, 1988). Any instructional
materials that embody a split attention e?ect cre-
ate cognitive load (Chandler & Sweller, 1992;
Sweller, Chandler, Tierney & Cooper, 1990). We
use the placement of explanations in a decision aid
to induce di?erential split attention e?ects,
thereby manipulating the level of cognitive load
experienced in using a decision aid.
1.3. Ability and motivation
A substantial body of cognitive psychology and
accounting research indicates that di?erences in
individual ability result in di?erential levels of
learning (e.g. Bonner & Walker, 1994; Horn, 1989;
Libby & Tan, 1994; Snow, 1989). These studies
also ®nd that general problem-solving ability is the
form of ability most closely related to learning.
Ability, however, does not act in isolation.
E?ort and ability have interactive e?ects on
learning and performance. Subjects with low-abil-
ity levels typically see few gains in performance as
a result of increased e?ort (Awasthi & Pratt,
1990). Cloyd (1997) examined the situations in
which e?ort can substitute for knowledge in a tax
search task. He found that e?ort can improve
performance for low-knowledge subjects in rela-
tively simple tasks, but not complex ones. This
result supports earlier work by Libby and Lipe
(1992), who found that for any individual, only
some cognitive processes (e.g. recall) can be
improved with e?ort, while other processes can
only be improved after more domain speci®c
knowledge is acquired. Finally, Cloyd showed
that, when compared to low-knowledge indivi-
duals, high-knowledge individuals achieve greater
increases in performance e?ectiveness for each unit
increase in e?ort across all tasks. In sum, increased
e?ort does not result in the same performance
e?ect for individuals of di?ering ability.
However, regardless of ability, Bernardo (1994)
found that subjects acquire schemata more rapidly
when they are told that learning is a task goal. His
results indicate that deliberate e?ort improves
288 J.M. Rose, C.J. Wolfe / Accounting, Organizations and Society 25 (2000) 285±306
learning. An analogical problem solving study by
Cummins (1992) found similar results. Subjects
learned faster when they were directed to pay
attention to the problem structure of analogous
problems.
1.4. Environment
Environment is manipulated in this research to
the extent that a hand calculation group using
traditional text-based materials is included as a
benchmark for comparison to the computerized
decision aid users.
1.5. Framework and hypotheses
As shown in Fig. 1, experience quality is de®ned
via the amount of cognitive load produced by a
decision aid. Cognitive load is modeled as a direct
determinant of learning, and increased cognitive
load is expected to result in decreased learning.
Additionally, cognitive load is modeled as a
determinant of the time spent learning (henceforth
referred to as learning e?ort duration). Todd and
Benbasat (1992, 1994) found that subjects tend to
view energy conservation as an important goal
when completing tasks with a computerized deci-
sion aid. Similarly, Glover et al. (1997) found that
subjects using a computerized decision aid spent
less time analyzing and reviewing a tax case than
did subjects who were not provided a decision aid.
Based on these ®ndings, it appears that automated
decision aids incent e?ort minimization strategies.
We expect that as the amount of cognitive load
produced by a decision aid increases, learning
e?ort duration will decrease, because the cognitive
load will make learning more dicult, thereby,
triggering e?ort minimization. Accordingly, our
framework indicates that the cognitive load pro-
duced by a decision aid can a?ect learning directly
and also indirectly through learning e?ort duration.
Libby and Tan (1994) and Awasthi and Pratt
(1990) propose that studies of the relationship
between experience quality and learning should
include controls for ability. This study follows those
guidelines. Our framework models problem-solving
ability, and it does so in a way not considered in
previous decision aid studies. We separate problem-
solving ability into two components: problem-
solving eciency and problem-solving e?ectiveness.
Eciency represents the time required to solve
problems, and e?ectiveness represents the faculty
to reach correct problem solutions.
3
Problem sol-
ving eciency is modeled as a determinant of
learning e?ort duration. Decision aid users with
high problem-solving eciency are not expected to
spend as much time learning from the aid as users
with lower problem-solving eciency. Our model-
ing of problem-solving e?ectiveness follows prior
literature. Individuals with high problem-solving
e?ectiveness should learn more from aid use than
individuals with lower problem-solving e?ectiveness
(Bonner & Walker, 1994; Libby & Tan, 1994). Also,
unit increases in learning e?ort duration should
result in greater performance improvements for
high problem-solving e?ectiveness subjects than
for low problem-solving e?ectiveness subjects
(Awasthi & Pratt, 1990; Cloyd, 1997).
As illustrated and noted in the framework
shown in Fig. 1, we propose the following list of
hypotheses (stated in alternative form).
Hypothesis 1. Subjects using decision aids that
promote higher split attention, i.e. produce more
cognitive load, will learn less than subjects using
decision aids that promote lower split attention.
Hypothesis 2. Increases in the amount of cognitive
load imposed by a decision aid will decrease the
user's learning e?ort duration.
Hypothesis 3. Decision aid users with higher pro-
blem-solving eciency will expend less learning
e?ort duration.
Hypothesis 4. Decision aid users who expend more
learning e?ort duration will learn more than users
who expend less learning e?ort duration.
Hypothesis 5. Decision aid users possessing higher
problem-solving e?ectiveness will learn more than
3
We know of no prior research that has investigated the role
of problem-solving eciency in decision aid or other similar
learning environments.
J.M. Rose, C.J. Wolfe / Accounting, Organizations and Society 25 (2000) 285±306 289
decision aid users possessing lower problem-sol-
ving e?ectiveness.
Hypothesis 6. Increases in learning e?ort duration
will result in greater learning for decision aid users
possessing high problem-solving e?ectiveness than
for decision aid users possessing lowproblem-solving
e?ectiveness.
2. Research method
2.1. Overview
Decision aids can be classi®ed into three major
categories: deterministic aids, decision support
systems, and expert systems (Abdolmohammadi,
1987; Messier & Hansen, 1987). Deterministic aids
are designed to produce complete solutions to
highly structured problems. In this research, we
use a deterministic decision aid to calculate tax
liabilities for an individual taxpayer in the United
States.
4
The experimental task requires computa-
tion of adjusted gross income, capital gains taxa-
tion, and taxable social security bene®ts. The task
did not require judgment on the part of the deci-
sion maker. This completely objective task
allowed us to measure knowledge acquisition after
aid use based upon unambiguous, ``correct'' pro-
cedures.
An outline of experimental operations is as fol-
lows. Subjects ®rst completed a knowledge pretest
to measure base levels of tax knowledge. Second,
they solved a set of Graduate Record Exam
(GRE) questions designed to measure two forms of
general problem-solving ability: problem-solving
e?ectiveness and problem-solving eciency. Next,
to familiarize subjects with the aid format, they
were trained on a mock decision aid devoid of
titles, explanations, or any tax-related material.
After that, subjects using di?erent forms of the
decision aid, and a hand calculation group, solved
three practice problems involving the calculation
of tax liabilities. Following the practice problems,
subjects' demographic information was collected,
and they performed a distracter task to clear
working memory of the tax rules. Finally, all sub-
jects completed a set of test questions without the
use of a decision aid or tax calculation instruc-
tions. The ®nal set of test questions was used to
measure the level of problem-type schema
acquired. Each subject completed the entire
experiment during one session in a graduate com-
puter lab.
2.2. Subjects
Subjects consisted of approximately 287 junior
and senior students enrolled in a 5-year account-
ing program at a large southwestern university in
the United States. In order to measure the acqui-
sition of schemata, student subjects were neces-
sary, because subjects must not have the relevant
schemata in place prior to administration of the
experiment. Use of student subjects and knowl-
edge pretests control for any prior knowledge.
Over 90% of the student subjects who participated
in this experiment interned with a Big 5 public
accounting ®rm (internships occurred after the
experiment). Historically, almost all interns have
accepted positions in public accounting. Our stu-
dent subjects appear reasonable surrogates for
entry-level accounting professionals.
To motivate students to exert e?ort during the
task, subjects were paid based upon their perfor-
mance. Subjects received $2.00 for each question
answered correctly in the ®nal test phase (up to a
maximum of $12.00). All subjects also received
credit in their courses for successful completion of
the entire experiment. Subjects were informed
before any experimental procedures began that
performance-based compensation was involved.
4
Fasttax and TurboTax represent examples of deterministic
decision aids that are regularly used by entry-level through
senior sta? accountants to prepare tax returns, and it is within
this aid environment that they acquire some of their tax
knowledge. To facilitate learning and explain the logic of the
underlying processes, these aids provide the rationale for their
calculations and/or refer to the tax code. Interviews with tax
managers in Big 5 accounting ®rms indicate that tax sta? spend
at least 50% of their time using decision aids, such as Fasttax
and Turbo Tax. Additionally, sta? accountants are expected to
learn from this experience. One of the contacted ®rms indicated
that they are currently developing their own tax preparation
decision aid with one of the express purposes being the
improvement of training through aid use.
290 J.M. Rose, C.J. Wolfe / Accounting, Organizations and Society 25 (2000) 285±306
However, they were not informed of the compen-
sation scheme until immediately before the
knowledge measurement phase of the experiment.
Description of the compensation scheme was
delayed in order to promote e?ort in all phases of
the experiment.
2.3. Experimental procedures
The experiment consisted of six phases: a pretest
phase, an ability measurement phase, an aid train-
ing phase, a schema acquisition phase, a distracter
phase, and a test phase. Subjects were randomly
Fig. 1. Framework for learning in decision aid environments.
J.M. Rose, C.J. Wolfe / Accounting, Organizations and Society 25 (2000) 285±306 291
assigned to one of four experimental treatments
prior to the ®rst phase. The pretest phase involved
cued recall of income tax rules. Pretest results were
used to control for any knowledge di?erences
existing prior to the experiment. In the second
phase, we measured individual problem-solving
e?ectiveness and problem-solving eciency. Gen-
eral problem-solving e?ectiveness was captured
based upon solution accuracy of eight GRE pro-
blems. The GRE problems were the same as those
used by Bonner and Walker (1994) and Bonner et
al. (1992) to measure general problem-solving
ability. Problem-solving eciency was measured
as the average standardized time spent on all GRE
problems answered correctly. Times were standar-
dized to control for the di?ering time requirements
of individual problems.
Subsequent to the ability measurement, subjects
were trained on the use of the decision aid. The aid
used for training was designed such that no tax
knowledge could be acquired through the use of
the aid. Subjects input information into the train-
ing aid in order to learn how to operate the aid
functions before problem solving began. Aid
training was necessary, because time consumed
solving problems with the decision aid measured
learning e?ort duration. Time spent learning to
use the aid would have contaminated this e?ort
measure. Aid training eliminated aid-learning
time from the time expended on solving pro-
blems with the decision aid.
After GRE problem solution and aid training,
subjects began the schema acquisition phase. This
phase involved the solution of a set of three prac-
tice problems (a sample practice problem appears
in the Appendix). Subjects were informed that
they had two goals while using the decision aids:
(1) to learn as much as possible about the calcula-
tion of tax liabilities, and (2) to answer the pro-
blems as accurately as possible. Subjects were
speci®cally instructed to learn the underlying tax
rules in order to promote the exertion of e?ort
towards knowledge acquisition. Bernardo (1994)
and Cummins (1992) both found that subjects
need to know that learning is a task goal for
schema acquisition to occur.
Treatments one through three used decision aids
with varying levels of split attention, and treatment
four performed all calculations by hand. Subjects
in treatment one received the decision aid with the
greatest inherent split attention e?ects. A decision
aid for calculating tax liabilities was presented on
one screen, and subjects were required to change
to another screen to view the tax calculation
instructions (see the Appendix). This aid most
closely approximates many popular aids currently
used in practice (e.g. Fasttax and TurboTax).
Subjects could switch screens at any time and as
many times as they wished. Switching screens
induces a heavy cognitive load, because subjects
must hold in working memory information gath-
ered from physically separate locations. Treatment
two subjects received a decision aid with the
instructions for tax computations on the same
screen as the aid. Subjects in this treatment were
still required to hold information in working
memory, but smaller chunks of information could
be examined at one time. In the third treatment,
subjects received a decision aid with the instruc-
tions integrated into the steps performed by the
aid. This treatment had the least inherent split
attention.
Treatment four completed the practice problems
by hand and received the same computer screen of
tax calculation instructions as the decision aid
users. The split attention level for treatment four is
not comparable to the decision aid treatments,
because this treatment did not receive a decision
aid. This no-aid treatment was used as a reference
group to compare learning between users and non-
users of computerized decision aids.
All decision aids were identical with the excep-
tion of the instruction placement. The instructions
themselves were also identical across treatments.
Treatment groups were physically separated to
prevent subjects from recognizing any di?erences
in treatment. To control for potential media
e?ects, all treatments performed the task on iden-
tical computers. Subjects were required to input
their answers for taxable income and tax liability
into speci®ed boxes. Subjects then received feed-
back, in the form of the correct answers, by
clicking on a ``CORRECT ANSWER'' button.
Feedback was necessary because it prevented
subjects from generating schemata for improper
solution strategies. The feedback was not explanatory,
292 J.M. Rose, C.J. Wolfe / Accounting, Organizations and Society 25 (2000) 285±306
but explanation was provided in the tax calcula-
tion instructions. Subjects were allowed the use of
scratch paper and identical calculators throughout
the practice problem phase. After checking
answers, subjects were allowed as much time as
they desired to reach the correct solution and
study/work the problems further. When subjects
were ready to begin a new problem, they clicked
on a ``DONE'' button. The total time spent on
each problem was captured by the computer.
This time was used to measure learning e?ort
duration expended on learning from the practice
problems.
Upon completion of the practice problems,
subjects were presented with a screen informing
them to turn in their scratch paper and to begin
the next phase. The fourth phase involved the
collection of demographic information and a dis-
tracter task designed to clear working memory.
Conway and Engle (1994) found that there are
di?erences in working memory capacity across
individuals, which result in di?erential recall abil-
ities. This study intended to measure the impact of
split attention on schema acquisition rather than
working memory retention and, therefore, work-
ing memory had to be cleared before the test
phase. Subjects were required to subtract the
number 13 from the number 467 for three con-
secutive repetitions. This simple task was shown
by Wickens et al. (1981) to be e?ective in clearing
the contents of working memory. In addition, the
demographic questionnaire itself acted as a dis-
tracter.
The test phase was designed to measure schema
acquisition. Schematic knowledge can be assessed
by having subjects (1) group problems into clus-
ters, (2) categorize problems after hearing only
part of the text, (3) solve problems when material
in the text is ambiguous, (4) identify information
in problems that is necessary and sucient for
solution, or (5) solve problems analogous to prac-
tice problems (Low & Over, 1992). This research
used the solution of analogous problems, due to
its rigor and ®t with our tax task. Solving the tax
problems requires both declarative knowledge of
the tax rules and procedural knowledge of their
application organized into problem-type schema
(Low & Over).
Completely unaided (i.e. no decision aid, no
instructions, and no notes) subjects were required
to solve ®ve problems that were analogous to the
practice problems and one transfer problem that
required rules covered in the practice problem
instructions, but not applied in a practice pro-
blem. The problems are de®ned as follows (a
sample problem appears in the Appendix).
1. A problem requiring knowledge of the con-
cepts of adjusted gross income, deductions,
and exemptions.
2. A problem requiring knowledge of the tax
liability computation rules for long-term
capital gains when taxpayers are in the 15 or
28% tax brackets (long-term capital gains
taxed as ordinary income).
3. A problem requiring knowledge of the tax
liability computation rules for long-term
capital gains when taxpayers are in tax
brackets above 28% (long-term capital gains
taxed at maximum marginal rate of 28%).
4. A problem requiring knowledge of the tax
computation rules for taxable social security
bene®ts when provisional income does not
exceed the ®rst base amount.
5. A problem requiring knowledge of the tax
computation rules for taxable social security
bene®ts when provisional income exceeds the
®rst base amount.
6. A problem requiring knowledge of the tax
computation rules for taxable social security
bene®ts when provisional income exceeds the
second base amount (transfer problem).
The six problems are arranged in a linear order
from simple to complex. To measure complexity,
we use the number of rules required for problem
solution (Low & Over, 1992; Sweller, 1988): pro-
blem one requires three rules, problem two
requires ®ve rules, problem three requires six rules,
problem four requires eight rules, problem ®ve
requires 11 rules, and problem six requires 12
rules. Problem one is required knowledge for all
other problems, but the remaining problems are
independent of one another with respect to
required knowledge (Low & Over). To control for
potential order e?ects in the test phase of the
J.M. Rose, C.J. Wolfe / Accounting, Organizations and Society 25 (2000) 285±306 293
experiment, the problems were randomly
ordered.
5
Rationale for the ordering of our test
phase problems rests on a ®nding by Low and
Over (1992) that indicates subjects fail to acquire
more complex schemata prior to the acquisition of
less complex schemata. To con®rm their results in
the tax domain, our six increasingly complex tax
problems were scored as either correct or incor-
rect. Fig. 2 shows the number of problems
answered correctly and incorrectly for all treat-
ments.
6
From Fig. 2, it is apparent that subjects
acquire this experiment's problem-type schema in
a linear fashion. That is, subjects who fail to
acquire simple schemata also fail to acquire more
complex schemata. The tree diagram in Fig. 2 can
be analyzed more formally using Guttman scalo-
gram analysis. The coecient of reproducibility
produced by Guttman scalogram analysis indi-
cates the predictability of the pattern of problem
solution. The coecient of reproducibility for the
six test phase problems is 0.989,
7
verifying that the
relative order of mastery in this group of tax pro-
blems is constant across subjects. This con®rmation
of Low and Over's ®nding is important, because it
indicates that subjects in this experiment with
higher scores on the test phase problems can solve
more complex tax problems than subjects with
lower scores, and likewise have acquired more
complex schemata.
5
All decision aids calculated the taxable social security ben-
e®ts before calculating tax on capital gains. This added strength
to the design because any recency e?ects would be in favor of
learning social security calculations before capital gains calcu-
lations. Schema theory predicts that capital gains schemata will
be acquired prior to social security schemata, because capital
gains schemata are less complex. Therefore, the acquisition of
capital gains schemata prior to social security schemata will not
be the result of a recency e?ect.
6
No scoring scale was used to produce Fig. 2. A problem
was scored as correct for this analysis only if there were no
errors in the solution.
7
The coecient of reproducibility was calculated using
Jackson's method (Maranell, 1974).
Fig. 2. Tree diagram of Test phase problems answered correctly.
294 J.M. Rose, C.J. Wolfe / Accounting, Organizations and Society 25 (2000) 285±306
2.4. Operational de®nitions of variables
2.4.1. Problem-solving performance (learning)
Problem-solving performance is measured by
the score obtained on the six problems in the test
phase of the experiment. The test phase problems
are completed without the use of a decision aid or
any tax calculation instructions after completing
the practice problems and a distracter task. The
test phase problems are scored 0±4 based on
accuracy and level of completion.
8
The total score,
which can range from 0 to 24, is used as a dependent
variable to represent problem-solving performance.
Di?erences in problem-solving performance represent
di?erences in the level of knowledge acquired,
because higher scores correspond to the successful
solution of increasingly complex problems.
2.4.2. Instruction integration (cognitive load)
The degree of instruction integration is varied
by di?erent placements of the tax calculation
instructions within each of the three decision aids.
The decision aid with instructions on a separate
screen has the lowest level of integration. The
decision aid with instructions on the same screen
has a moderate level of integration. The decision
aid with integrated instructions (i.e. instructions
adjoining the decision aid queries) has the greatest
level of integration.
2.4.3. Learning e?ort duration
Learning e?ort duration is captured by the time
spent working the three practice problems. The
time required to learn how to operate the decision
aid is removed from this measure by training sub-
jects on a decision aid devoid of any tax informa-
tion or instructions before they worked the
practice problems. All three decision aids can be
used to calculate tax liabilities in the same amount
of time. Di?erences in time spent working the
practice problems relate to di?erences in e?ort
duration directed towards learning the aid's
underlying calculation logic.
2.4.4. Problem-solving e?ectiveness
Problem-solving e?ectiveness is measured by the
number of correctly answered GRE questions.
There are eight GRE questions. Therefore, the
variable ranges from 0 to 8.
2.4.5. Problem-solving eciency
Problem-solving eciency is calculated based on
the time subjects spent solving correctly answered
GRE questions. The time required for each of the
eight questions varies and, therefore, a standar-
dized score for eciency is constructed. Each
subject's eciency is calculated as follows:
?Æ?TGE
ij
À MTGRE
i
?=SDTGRE
i
? Ã GRE
ij
??=
GRETOTAL
i
where, TGE
ij
=time spent on GRE question i for
subject j, MTGRE
i
=mean time spent by all sub-
jects on GRE question i, SDTGRE
i
=standard
deviation of time spent on GRE question i,
GRE
ij
=score (0,1) received by subject i on GRE
question j, GRETOTAL
i
=total score on GRE
questions (0±8) for subject i.
This formula yields an eciency score based on
the average deviation from mean solution times
for each subject. Lower score values indicate
greater problem-solving eciency.
9
3. Results
3.1. Preliminary statistics
A total of 287 subjects participated in the
experiment. Thirty-four subjects were removed
8
There was no need to compute the interrater reliability as
the grading scale was completely objective. The grader was
unaware of the treatments associated with each problem set.
9
Only times on GRE problems answered correctly were
used to calculate problem-solving eciency, because the inclu-
sion of problems answered incorrectly could contaminate the
eciency measure. However, it is also possible that subjects
who skip or gloss-over all problems could receive very high
eciency scores using our eciency metric. Therefore, two
alternative measures of problem-solving eciency that included
GRE questions not answered correctly were calculated. The
®rst alternative eciency measure was a similar standardized
score, but the score included times on all eight GRE questions.
The second and simplest measure was calculated by dividing
the total time spent solving GRE questions by the total number
of questions. Reported results do not change substantively
using either alternative eciency measure.
J.M. Rose, C.J. Wolfe / Accounting, Organizations and Society 25 (2000) 285±306 295
from the analyses because of prior knowledge or
failure to complete all aspects of the experiment.
10
Therefore, all analyses were conducted using the
remaining 253 subjects. The average age and GPA
of the subjects were 21 and 3.48, respectively.
Descriptive statistics and correlations are pre-
sented in Table 1.
3.2. Instruction integration
Hypothesis one proposes that subjects using deci-
sion aids with lower degrees of instruction integration
will have lower problem-solving performance (i.e.
increased cognitive load will reduce learning). This is
tested with an analysis of covariance (ANCOVA)
model using the instruction integration treatments
as a dependent variable, and the covariates, learning
e?ort duration, problem-solving e?ectiveness, and
problem-solving eciency, act as controls on the
independent variable, problem-solving performance.
As shown in Table 2, the ANCOVA model is statis-
tically signi®cant at the 0.0001 level.
11
Two of the
covariates, learning e?ort duration and problem-
solving e?ectiveness, are statistically signi®cant at
the 0.0001 level, and the treatment variable, degree
of instruction integration, is signi®cant at the
0.0001 level.
12
The Student±Newman±Keuls (SNK)
test for di?erences between least square means
indicates that after controlling for learning e?ort
duration and problem-solving e?ectiveness and
eciency, subjects in the integrated instructions
treatment learn more than subjects in the same
screen or separate screen treatments. These results
provide strong support for hypothesis one.
13
Table 1
Preliminary statistics
a
Variable N Mean Deviation Minimum Maximum
Panel A: descriptive statistics
Problem-solving e?ectiveness 253.00 5.95 1.23 2.00 8.00
Problem-solving eciency 253.00 À0.0032 0.55 À0.96 3.97
Problem-solving performance score 253.00 10.4200 5.55 0.00 24.00
Learning e?ort duration 253.00 856.62 464.46 329.00 3003.00
Problem-solving
performance
score
Problem-solving
e?ectiveness
Problem-solving
eciency
Learning
e?ort
duration
Panel B: correlation matrix
Problem-solving performance score 1.000 0.251
Ã
À0.080 0.454
Ã
Problem-solving e?ectiveness 1.000 À0.206
Ã
0.011
Problem-solving eciency 1.000 0.129
Learning e?ort duration 1.000
a
*Signi®cant at 0.05 level.
10
Any subject who had existing knowledge of the tax mate-
rial examined in our experiment was removed from the ana-
lyses. Prior knowledge was measured with a cued recall pretest.
11
Levene's test for homogeneity of variances indicates that
error variances were not equal across treatments. ANOVA,
however, is robust for violations of the homogeneity of var-
iance assumption when sample sizes are approximately equal.
To validate the results using a non-parametric procedure that
does not rely on a homogeneity of variance assumption, the
Kruskal±Wallis procedure and mean comparisons were con-
ducted using rank scores. Results for statistically signi®cant
di?erences were unchanged.
12
The analyses of treatment di?erences were repeated using
the ordinal ranking of the most complex problem solved as the
dependent variable. This measure of problem-solving perfor-
mance does not rely on any form of objective scoring. No
qualitative di?erences in results were found.
13
All analyses were repeated without including the score for
problem six, because problem six was not analogous to the
practice problems. No di?erences in results were found.
296 J.M. Rose, C.J. Wolfe / Accounting, Organizations and Society 25 (2000) 285±306
An important result stems from the no-aid
treatment group that performed all calculations by
hand. Consistent with prior research (Glover et
al., 1997; Murphy, 1990), the hand calculation
group's problem-solving performance was higher
than any decision aid treatment group. However,
when comparing the hand calculation group to
subjects using the integrated instructions decision
aid (i.e. the aid that produced the least cognitive
load), we found that the hand calculation group
had problem-solving performance scores that were
22% higher, but at cost of a 112% increase in
learning e?ort duration. When individual di?er-
ences in learning e?ort duration and problem-solving
e?ectiveness were controlled, problem-solving
performance di?erences were insigni®cant between
subjects doing hand calculation and subjects using
the integrated instructions decision aid. This ®nding
indicates that individuals of equal ability putting
forth equivalent e?ort can learn-by-doing equally
well, regardless of whether the process involves a
decision aid or not; as long as the decision aid
does not produce a large amount of cognitive
load.
3.3. Learning e?ort duration
The second hypothesis proposes that learning
e?ort duration will decrease as the degree of
instruction integration in the decision aid decrea-
ses (i.e. heightened cognitive load will reduce
learning e?ort duration). Table 3 displays
ANCOVA results and SNK least square mean
comparisons for learning e?ort duration while
controlling for problem-solving e?ectiveness and
eciency. Overall, the no aid treatment had higher
learning e?ort duration than any other treatment,
but no statistically signi®cant di?erence is found
between the decision aid treatments. The SNK
mean separation procedure does not support the
second hypothesis. Decreases in instruction inte-
gration did not lead to decreases in learning e?ort
duration. Given that results support the null for
hypothesis two, drawing any strong conclusion is
not feasible.
The third hypothesis states that decision aid
users with higher problem-solving eciency will
expend less learning e?ort duration. The
ANCOVA model in Table 3 indicates that pro-
blem-solving eciency is statistically signi®cant at
Table 2.
Analysis of problem-solving performance
ANCOVA and least square mean comparisons
a
Source Sum of square df Mean square F Signi®cance
Problem-solving e?ectiveness (covariate) 520.398 1 520.398 26.796 0.000
Learning e?ort duration (covariate) 273.001 1 273.001 14.057 0.000
Problem-solving eciency (covariate) 40.134 1 40.134 2.067 0.152
Between groups 856.126 3 285.375 14.694 0.000
Within groups 4777.531 246 19.421
Total 252
Di?erences in least square treatment means
Treatment LS mean Separate Same Integrated No aid
Separate screen 7.31 ± 2.69
Ã
4.97
**
4.98
**
Same Screen 9.99 ± 2.29
Ã
2.30
Ã
Integrated 12.28 ± 0.01
No aid 12.29 ±
a
R-square=0.385.
Ã
Signi®cant at p
Accounting ®rms are intensifying their reliance on experiential learning, and experience increasingly involves the use
of computerized decision aids [Messier, W. (1995) Research in and development of audit decision aids. In R. H. Ashton
& A. H. Ashton, Judgment and decision making in accounting and auditing (pp. 207±230). New York: Cambridge
University Press]. Accountants are expected to learn from automated decision aid use, because the aids are not always
available when dealing with the aid's topical matter, and the knowledge inherent in the aid is needed for competency on
broader issues. To facilitate knowledge acquisition and explain the logic of the underlying processes, computerized
decision aids provide the rationale for their calculations in the form of online explanations. We study how the location
of explanations in a computerized decision aid aects learning from its use. Speci®cally, this research extends the
existing literature by using a framework for the study of learning from decision aid use and by using cognitive load
theory to explain the failure of certain decision aid design alternatives to promote learning.
The e?ects of system design alternatives on the acquisition
of tax knowledge from a computerized tax decision aid
Jacob M. Rose
a,
*, Christopher J. Wolfe
b
a
Department of Accounting, Bryant College, 1150 Douglas Pike, Smith®eld, RI 02917, USA
b
Department of Accounting, Lowry Mays College & Graduate School of Business, Texas A&M University, College Station,
TX 77843-4353, USA
Abstract
Accounting ®rms are intensifying their reliance on experiential learning, and experience increasingly involves the use
of computerized decision aids [Messier, W. (1995) Research in and development of audit decision aids. In R. H. Ashton
& A. H. Ashton, Judgment and decision making in accounting and auditing (pp. 207±230). New York: Cambridge
University Press]. Accountants are expected to learn from automated decision aid use, because the aids are not always
available when dealing with the aid's topical matter, and the knowledge inherent in the aid is needed for competency on
broader issues. To facilitate knowledge acquisition and explain the logic of the underlying processes, computerized
decision aids provide the rationale for their calculations in the form of online explanations. We study how the location
of explanations in a computerized decision aid a?ects learning from its use. Speci®cally, this research extends the
existing literature by using a framework for the study of learning from decision aid use and by using cognitive load
theory to explain the failure of certain decision aid design alternatives to promote learning. We de®ne learning as the
acquisition of problem-type schemata, and an experiment is performed in which cognitive load is manipulated by the
placement of explanations in a computerized tax decision aid to determine its e?ect on schema acquisition. Schemata
are general knowledge structures used for basic comprehension, and cognitive load refers to the burden placed on
working memory when acquiring schemata. We ®nd that increased cognitive load produced by the location of expla-
nations in a decision aid leads to reduced schema acquisition. Our results indicate that when explanations in a com-
puterized decision aid are integrated into its problem solving steps, cognitive load is reduced and users acquire more
knowledge from aid use. This appears to be an important design consideration for accounting ®rms buying or building
computerized decision aids. # 2000 Elsevier Science Ltd. All rights reserved.
This study investigates the determinants of
knowledge acquisition from the use of an auto-
mated tax decision aid. Under the rationale of
eciency and e?ectiveness in decision making,
computer-based decision aids are commonly used
in public accounting (Brown & Eining, 1997;
Messier, 1995). However, assistance in making
decisions is not the only expected function of these
aids. It has been conjectured that experience gar-
nered while using decision aids also promotes
knowledge acquisition, because the aid should
0361-3682/00/$ - see front matter # 2000 Elsevier Science Ltd. All rights reserved.
PI I : S0361- 3682( 99) 00048- 3
Accounting, Organizations and Society 25 (2000) 285±306
www.elsevier.com/locate/aos
* Corresponding author.
E-mail address: [email protected] (J.M. Rose), cjwolfe@
tamu.edu (C.J. Wolfe).
provide an illustration of proper problem solving
method, explanation of the method, and outcome
feedback (Ashton & Willingham, 1988; Pei, Stein-
bart & Reneau, 1994). Pragmatically, experiential
learning from the use of an automated decision
aid is important for at least two reasons: (1) The
aids will not always be conveniently available, and
accounting practitioners often deal with client
scenarios on an ad hoc basis; and (2) The base
knowledge inherent in decision aids must be part
of an accounting professional's repertoire, because
as sta? rise to managerial positions they must be
able to evaluate the output of decision aids in a
broader context.
1
Knowledge has been shown to be a functional
determinant of decision performance (Bonner &
Lewis, 1990; Libby & Tan, 1994). Therefore,
learning from using an automated decision aid is
important to decision performance when making
decisions inside an aid's domain without the use of
the aid and when evaluating the ecacy of an aid's
output. To understand the development of exper-
tise in environments characterized by the use of
automated decision aids, an important implication
is that a detailed understanding of knowledge
acquisition from using such aids is needed ®rst.
Research on learning from computerized deci-
sion aid use has focused on two general questions:
(1) How does experiential learning of computer-
ized decision aid users di?er from hand calculation
groups using traditional text-based materials; and
(2) Can user or system attributes be manipulated
to enhance learning from computerized decision
aid use? Research results on learning di?erences
between computerized decision aid users and hand
calculation groups indicate that hand calculation
treatments outperform aid users when given tra-
ditional text-based materials that facilitate a com-
plete solution to the experimental problems
(Glover, Prawitt & Spilker, 1997; Murphy, 1990).
2
While these ®ndings are compelling, the bene®ts of
decision consistency, eciency, and documenta-
tion apparently outweigh the sub-optimal learning
experience of automated decision aid use, because
accounting ®rms continue to make heavy use of
such aids. Therefore, the more critical question is
the second: Can anything be done to increase
experiential learning when using computerized
decision aids?
Approaching this question from the side of the
decision aid user, the earliest line of research
addressed the possibility that mismatches between
users' knowledge organization and the underlying
structure of the decision aid led to learning de®cits
(Frederick, 1991; Pei & Reneau, 1990; Ricchiute,
1992). These studies found that knowledge acqui-
sition was improved when decision aid structures
matched the knowledge structures of their users.
However, any strategy based on these ®ndings
shifts much of the training burden away from the
experience of using the decision aid.
Modifying the design of a decision aid to
enhance its training capability is superior to train-
ing users on the knowledge structure of a decision
aid, because complete experiential learning
through automated decision aid use is more e-
cient. The design feature inherent in a computer-
ized decision aid to assist learning is the aid's
explanation facility (i.e. a software device that
explains calculation logic). Early research com-
paring the presence or absence of explanations in
a decision aid found explanations inconsequential
to learning (Eining & Dorr, 1991; Murphy, 1990).
Additionally, Steinbart and Accola (1994) found
that more elaborate explanations did not promote
a greater level of learning, and no learning e?ect
was identi®ed for the di?ering placement of
explanations within a decision aid (Mot, 1994;
Odom & Dorr, 1995).
The current study extends the existing literature
by focusing on explanation placement within a
1
Demonstrating ®rm emphasis on experiential learning,
Price Waterhouse Coopers has shifted a signi®cant component
of their tax training to ``structured work assignments in the
oce.'' Deloitte and Touche has also increased its emphasis on
learning through experience. Employee manuals stress that in
today's quickly changing ®nancial environment, on-the-job
training ``is essential to maintain the level of competence
necessary to render excellent service.''
2
While Fedorowicz, Oz and Berger (1992) and Eining and
Dorr (1991) found that computerized decision aid users learned
more than hand calculation groups, equivalency di?erences
existed in the decision support tools for the hand calculation
and computerized treatments.
286 J.M. Rose, C.J. Wolfe / Accounting, Organizations and Society 25 (2000) 285±306
computerized decision aid using theoretical fra-
meworks taken from the educational psychology
and accounting literatures. More precisely, we use
cognitive load theory to explain di?erences in
schema acquisition due to the location of expla-
nations in a computerized decision aid. Schemata
are general knowledge structures used for basic
comprehension and can be de®ned as ``cognitive
constructs that permit problem-solvers to recog-
nize a problem as belonging to a speci®c category
requiring particular moves for completion'' (Tar-
mizi & Sweller, 1988). Cognitive load refers to the
burden placed on working memory when acquir-
ing schemata (Sweller, 1988). In this study, we
conduct an experiment where levels of cognitive
load produced by decision aids are manipulated
by varying the placement of explanations in a
decision aid, and a knowledge acquisition frame-
work is used to investigate the e?ects of cognitive
load on learning from a decision aid.
The framework models the direct and indirect
e?ects of the cognitive load produced by a deci-
sion aid on its user's capacity to learn from the
experience of using the aid, while accounting for
the problem-solving ability of the aid user and the
amount of time spent learning from aid use. A
number of ®ndings emanate from our framework-
based analysis, but the most salient are as follows.
Cognitive load plays an important role when
learning from a computerized decision aid: users
of decision aids producing the least cognitive load
learned the most. We also ®nd that subjects per-
forming hand calculation with text-based instruc-
tions scored 22% higher on our learning
performance measure than did subjects using the
decision aid that produced the lowest cognitive
load, but the hand calculation group spent more
than double the amount of time in the learning
process. When di?erences in time spent learning
and problem-solving ability are statistically con-
trolled, we show that subjects using the decision
aid producing the lowest cognitive load learned
equivalently to subjects performing hand calcula-
tion with text-based instructions.
It appears important for system designers and
accounting ®rms to consider the cognitive load
imposed by a decision aid. Accounting tasks have
speci®c processing requirements, and ®rms have
automated many of these tasks with decision aids.
This research indicates that the placement of
explanations within these aids signi®cantly a?ects
their user's ability to learn from the aid. The
remainder of the paper describes the framework
and hypotheses, followed by a description of the
methods and results. The ®nal section includes
discussion and conclusions, as well as limitations
and suggestions for future research.
1. Framework and Hypotheses
Eq. (1) represents Libby's (1995) model of func-
tional determinants for knowledge:
Knowledge ? f ?Experience; Ability;
Motivation; Environment?
?1?
This research follows that structure de®ning the
constructs and framework as follows.
1.1. Knowledge
Schemata represent the structure and organiza-
tion of knowledge in long-term memory. Psychol-
ogy research indicates that experts have schemata
that allow them to organize and retrieve speci®c
information, while novices lack these schemata.
Similarly, accountants develop detailed schemata
for recognizing and solving accounting problems.
Weber (1980) validated the existence of accounting
schemata in a free recall task requiring auditors to
recall information technology (IT) controls. He
found that experienced IT auditors had higher
levels of cue clustering than students, suggesting
that experienced auditors possessed schemata
which students lacked. Libby (1985) discovered
that possible errors identi®ed by experienced
auditors during the analytical review process
occurred more frequently in practice than the
errors identi®ed by novices. Experts had ®nancial
statement error schemata that allowed them to
recognize reasonable errors. Finally, Frederick
(1991) demonstrated that experienced auditors
organize internal control knowledge based on
transaction ¯ows while novices organize by internal
J.M. Rose, C.J. Wolfe / Accounting, Organizations and Society 25 (2000) 285±306 287
control objectives. The experienced auditors had
schemata that organized their knowledge into a
structure useful for problem solving in the audit-
ing environment. These accounting studies have
generally assumed the existence of schemata, but
have not examined how and under what circum-
stances they are acquired.
In this study, learning is characterized as the
acquisition of schemata. We examine problem-
type schemata which can be de®ned as ``a cogni-
tive construct that permits problem-solvers to
recognize a problem as belonging to a speci®c
category requiring particular moves for comple-
tion'' (Tarmizi & Sweller, 1988).
1.2. Experience
Experience is the treatment in this study and is
de®ned as the quality of an encounter with a
computerized decision aid. The encounter pro-
vides both ®rst-hand and second-hand experiential
learning opportunities. Completion of a problem
using the decision aid represents a ®rst hand
encounter, and reading the explanation o?ered by
the decision aid can be construed as a second-
hand encounter, if it educates the decision aid user
in a broader context. In this research, we manip-
ulate the quality of experience by manipulating the
location of a decision aid's explanations.
Changing the location of explanations in a
decision aid should induce cognitive load and
a?ect schema acquisition. Cognitive load stems
from the interrelationship between working mem-
ory and long-term memory. When acquiring sche-
mata, working memory is used to actively process
new information before it is stored in long-term
memory. Therefore, working memory de®nes the
limits of schema acquisition as the processing
bottleneck (Mousavi, Low & Sweller, 1995; Swel-
ler, 1988, 1993). In experiential learning, problem
solving space in working memory crowds out that
needed for schema acquisition. This burden on
working memory is referred to as cognitive load.
We manipulate the level of cognitive load
through split attention e?ects. Attention is split
when problem solvers must hold information from
one source in working memory while attempting
to evaluate information from one or more other
sources. Splitting attention reduces the memory
available for acquiring schemata, because it
requires information to be held in working mem-
ory and then combined with other information
(Tarmizi & Sweller, 1988). Any instructional
materials that embody a split attention e?ect cre-
ate cognitive load (Chandler & Sweller, 1992;
Sweller, Chandler, Tierney & Cooper, 1990). We
use the placement of explanations in a decision aid
to induce di?erential split attention e?ects,
thereby manipulating the level of cognitive load
experienced in using a decision aid.
1.3. Ability and motivation
A substantial body of cognitive psychology and
accounting research indicates that di?erences in
individual ability result in di?erential levels of
learning (e.g. Bonner & Walker, 1994; Horn, 1989;
Libby & Tan, 1994; Snow, 1989). These studies
also ®nd that general problem-solving ability is the
form of ability most closely related to learning.
Ability, however, does not act in isolation.
E?ort and ability have interactive e?ects on
learning and performance. Subjects with low-abil-
ity levels typically see few gains in performance as
a result of increased e?ort (Awasthi & Pratt,
1990). Cloyd (1997) examined the situations in
which e?ort can substitute for knowledge in a tax
search task. He found that e?ort can improve
performance for low-knowledge subjects in rela-
tively simple tasks, but not complex ones. This
result supports earlier work by Libby and Lipe
(1992), who found that for any individual, only
some cognitive processes (e.g. recall) can be
improved with e?ort, while other processes can
only be improved after more domain speci®c
knowledge is acquired. Finally, Cloyd showed
that, when compared to low-knowledge indivi-
duals, high-knowledge individuals achieve greater
increases in performance e?ectiveness for each unit
increase in e?ort across all tasks. In sum, increased
e?ort does not result in the same performance
e?ect for individuals of di?ering ability.
However, regardless of ability, Bernardo (1994)
found that subjects acquire schemata more rapidly
when they are told that learning is a task goal. His
results indicate that deliberate e?ort improves
288 J.M. Rose, C.J. Wolfe / Accounting, Organizations and Society 25 (2000) 285±306
learning. An analogical problem solving study by
Cummins (1992) found similar results. Subjects
learned faster when they were directed to pay
attention to the problem structure of analogous
problems.
1.4. Environment
Environment is manipulated in this research to
the extent that a hand calculation group using
traditional text-based materials is included as a
benchmark for comparison to the computerized
decision aid users.
1.5. Framework and hypotheses
As shown in Fig. 1, experience quality is de®ned
via the amount of cognitive load produced by a
decision aid. Cognitive load is modeled as a direct
determinant of learning, and increased cognitive
load is expected to result in decreased learning.
Additionally, cognitive load is modeled as a
determinant of the time spent learning (henceforth
referred to as learning e?ort duration). Todd and
Benbasat (1992, 1994) found that subjects tend to
view energy conservation as an important goal
when completing tasks with a computerized deci-
sion aid. Similarly, Glover et al. (1997) found that
subjects using a computerized decision aid spent
less time analyzing and reviewing a tax case than
did subjects who were not provided a decision aid.
Based on these ®ndings, it appears that automated
decision aids incent e?ort minimization strategies.
We expect that as the amount of cognitive load
produced by a decision aid increases, learning
e?ort duration will decrease, because the cognitive
load will make learning more dicult, thereby,
triggering e?ort minimization. Accordingly, our
framework indicates that the cognitive load pro-
duced by a decision aid can a?ect learning directly
and also indirectly through learning e?ort duration.
Libby and Tan (1994) and Awasthi and Pratt
(1990) propose that studies of the relationship
between experience quality and learning should
include controls for ability. This study follows those
guidelines. Our framework models problem-solving
ability, and it does so in a way not considered in
previous decision aid studies. We separate problem-
solving ability into two components: problem-
solving eciency and problem-solving e?ectiveness.
Eciency represents the time required to solve
problems, and e?ectiveness represents the faculty
to reach correct problem solutions.
3
Problem sol-
ving eciency is modeled as a determinant of
learning e?ort duration. Decision aid users with
high problem-solving eciency are not expected to
spend as much time learning from the aid as users
with lower problem-solving eciency. Our model-
ing of problem-solving e?ectiveness follows prior
literature. Individuals with high problem-solving
e?ectiveness should learn more from aid use than
individuals with lower problem-solving e?ectiveness
(Bonner & Walker, 1994; Libby & Tan, 1994). Also,
unit increases in learning e?ort duration should
result in greater performance improvements for
high problem-solving e?ectiveness subjects than
for low problem-solving e?ectiveness subjects
(Awasthi & Pratt, 1990; Cloyd, 1997).
As illustrated and noted in the framework
shown in Fig. 1, we propose the following list of
hypotheses (stated in alternative form).
Hypothesis 1. Subjects using decision aids that
promote higher split attention, i.e. produce more
cognitive load, will learn less than subjects using
decision aids that promote lower split attention.
Hypothesis 2. Increases in the amount of cognitive
load imposed by a decision aid will decrease the
user's learning e?ort duration.
Hypothesis 3. Decision aid users with higher pro-
blem-solving eciency will expend less learning
e?ort duration.
Hypothesis 4. Decision aid users who expend more
learning e?ort duration will learn more than users
who expend less learning e?ort duration.
Hypothesis 5. Decision aid users possessing higher
problem-solving e?ectiveness will learn more than
3
We know of no prior research that has investigated the role
of problem-solving eciency in decision aid or other similar
learning environments.
J.M. Rose, C.J. Wolfe / Accounting, Organizations and Society 25 (2000) 285±306 289
decision aid users possessing lower problem-sol-
ving e?ectiveness.
Hypothesis 6. Increases in learning e?ort duration
will result in greater learning for decision aid users
possessing high problem-solving e?ectiveness than
for decision aid users possessing lowproblem-solving
e?ectiveness.
2. Research method
2.1. Overview
Decision aids can be classi®ed into three major
categories: deterministic aids, decision support
systems, and expert systems (Abdolmohammadi,
1987; Messier & Hansen, 1987). Deterministic aids
are designed to produce complete solutions to
highly structured problems. In this research, we
use a deterministic decision aid to calculate tax
liabilities for an individual taxpayer in the United
States.
4
The experimental task requires computa-
tion of adjusted gross income, capital gains taxa-
tion, and taxable social security bene®ts. The task
did not require judgment on the part of the deci-
sion maker. This completely objective task
allowed us to measure knowledge acquisition after
aid use based upon unambiguous, ``correct'' pro-
cedures.
An outline of experimental operations is as fol-
lows. Subjects ®rst completed a knowledge pretest
to measure base levels of tax knowledge. Second,
they solved a set of Graduate Record Exam
(GRE) questions designed to measure two forms of
general problem-solving ability: problem-solving
e?ectiveness and problem-solving eciency. Next,
to familiarize subjects with the aid format, they
were trained on a mock decision aid devoid of
titles, explanations, or any tax-related material.
After that, subjects using di?erent forms of the
decision aid, and a hand calculation group, solved
three practice problems involving the calculation
of tax liabilities. Following the practice problems,
subjects' demographic information was collected,
and they performed a distracter task to clear
working memory of the tax rules. Finally, all sub-
jects completed a set of test questions without the
use of a decision aid or tax calculation instruc-
tions. The ®nal set of test questions was used to
measure the level of problem-type schema
acquired. Each subject completed the entire
experiment during one session in a graduate com-
puter lab.
2.2. Subjects
Subjects consisted of approximately 287 junior
and senior students enrolled in a 5-year account-
ing program at a large southwestern university in
the United States. In order to measure the acqui-
sition of schemata, student subjects were neces-
sary, because subjects must not have the relevant
schemata in place prior to administration of the
experiment. Use of student subjects and knowl-
edge pretests control for any prior knowledge.
Over 90% of the student subjects who participated
in this experiment interned with a Big 5 public
accounting ®rm (internships occurred after the
experiment). Historically, almost all interns have
accepted positions in public accounting. Our stu-
dent subjects appear reasonable surrogates for
entry-level accounting professionals.
To motivate students to exert e?ort during the
task, subjects were paid based upon their perfor-
mance. Subjects received $2.00 for each question
answered correctly in the ®nal test phase (up to a
maximum of $12.00). All subjects also received
credit in their courses for successful completion of
the entire experiment. Subjects were informed
before any experimental procedures began that
performance-based compensation was involved.
4
Fasttax and TurboTax represent examples of deterministic
decision aids that are regularly used by entry-level through
senior sta? accountants to prepare tax returns, and it is within
this aid environment that they acquire some of their tax
knowledge. To facilitate learning and explain the logic of the
underlying processes, these aids provide the rationale for their
calculations and/or refer to the tax code. Interviews with tax
managers in Big 5 accounting ®rms indicate that tax sta? spend
at least 50% of their time using decision aids, such as Fasttax
and Turbo Tax. Additionally, sta? accountants are expected to
learn from this experience. One of the contacted ®rms indicated
that they are currently developing their own tax preparation
decision aid with one of the express purposes being the
improvement of training through aid use.
290 J.M. Rose, C.J. Wolfe / Accounting, Organizations and Society 25 (2000) 285±306
However, they were not informed of the compen-
sation scheme until immediately before the
knowledge measurement phase of the experiment.
Description of the compensation scheme was
delayed in order to promote e?ort in all phases of
the experiment.
2.3. Experimental procedures
The experiment consisted of six phases: a pretest
phase, an ability measurement phase, an aid train-
ing phase, a schema acquisition phase, a distracter
phase, and a test phase. Subjects were randomly
Fig. 1. Framework for learning in decision aid environments.
J.M. Rose, C.J. Wolfe / Accounting, Organizations and Society 25 (2000) 285±306 291
assigned to one of four experimental treatments
prior to the ®rst phase. The pretest phase involved
cued recall of income tax rules. Pretest results were
used to control for any knowledge di?erences
existing prior to the experiment. In the second
phase, we measured individual problem-solving
e?ectiveness and problem-solving eciency. Gen-
eral problem-solving e?ectiveness was captured
based upon solution accuracy of eight GRE pro-
blems. The GRE problems were the same as those
used by Bonner and Walker (1994) and Bonner et
al. (1992) to measure general problem-solving
ability. Problem-solving eciency was measured
as the average standardized time spent on all GRE
problems answered correctly. Times were standar-
dized to control for the di?ering time requirements
of individual problems.
Subsequent to the ability measurement, subjects
were trained on the use of the decision aid. The aid
used for training was designed such that no tax
knowledge could be acquired through the use of
the aid. Subjects input information into the train-
ing aid in order to learn how to operate the aid
functions before problem solving began. Aid
training was necessary, because time consumed
solving problems with the decision aid measured
learning e?ort duration. Time spent learning to
use the aid would have contaminated this e?ort
measure. Aid training eliminated aid-learning
time from the time expended on solving pro-
blems with the decision aid.
After GRE problem solution and aid training,
subjects began the schema acquisition phase. This
phase involved the solution of a set of three prac-
tice problems (a sample practice problem appears
in the Appendix). Subjects were informed that
they had two goals while using the decision aids:
(1) to learn as much as possible about the calcula-
tion of tax liabilities, and (2) to answer the pro-
blems as accurately as possible. Subjects were
speci®cally instructed to learn the underlying tax
rules in order to promote the exertion of e?ort
towards knowledge acquisition. Bernardo (1994)
and Cummins (1992) both found that subjects
need to know that learning is a task goal for
schema acquisition to occur.
Treatments one through three used decision aids
with varying levels of split attention, and treatment
four performed all calculations by hand. Subjects
in treatment one received the decision aid with the
greatest inherent split attention e?ects. A decision
aid for calculating tax liabilities was presented on
one screen, and subjects were required to change
to another screen to view the tax calculation
instructions (see the Appendix). This aid most
closely approximates many popular aids currently
used in practice (e.g. Fasttax and TurboTax).
Subjects could switch screens at any time and as
many times as they wished. Switching screens
induces a heavy cognitive load, because subjects
must hold in working memory information gath-
ered from physically separate locations. Treatment
two subjects received a decision aid with the
instructions for tax computations on the same
screen as the aid. Subjects in this treatment were
still required to hold information in working
memory, but smaller chunks of information could
be examined at one time. In the third treatment,
subjects received a decision aid with the instruc-
tions integrated into the steps performed by the
aid. This treatment had the least inherent split
attention.
Treatment four completed the practice problems
by hand and received the same computer screen of
tax calculation instructions as the decision aid
users. The split attention level for treatment four is
not comparable to the decision aid treatments,
because this treatment did not receive a decision
aid. This no-aid treatment was used as a reference
group to compare learning between users and non-
users of computerized decision aids.
All decision aids were identical with the excep-
tion of the instruction placement. The instructions
themselves were also identical across treatments.
Treatment groups were physically separated to
prevent subjects from recognizing any di?erences
in treatment. To control for potential media
e?ects, all treatments performed the task on iden-
tical computers. Subjects were required to input
their answers for taxable income and tax liability
into speci®ed boxes. Subjects then received feed-
back, in the form of the correct answers, by
clicking on a ``CORRECT ANSWER'' button.
Feedback was necessary because it prevented
subjects from generating schemata for improper
solution strategies. The feedback was not explanatory,
292 J.M. Rose, C.J. Wolfe / Accounting, Organizations and Society 25 (2000) 285±306
but explanation was provided in the tax calcula-
tion instructions. Subjects were allowed the use of
scratch paper and identical calculators throughout
the practice problem phase. After checking
answers, subjects were allowed as much time as
they desired to reach the correct solution and
study/work the problems further. When subjects
were ready to begin a new problem, they clicked
on a ``DONE'' button. The total time spent on
each problem was captured by the computer.
This time was used to measure learning e?ort
duration expended on learning from the practice
problems.
Upon completion of the practice problems,
subjects were presented with a screen informing
them to turn in their scratch paper and to begin
the next phase. The fourth phase involved the
collection of demographic information and a dis-
tracter task designed to clear working memory.
Conway and Engle (1994) found that there are
di?erences in working memory capacity across
individuals, which result in di?erential recall abil-
ities. This study intended to measure the impact of
split attention on schema acquisition rather than
working memory retention and, therefore, work-
ing memory had to be cleared before the test
phase. Subjects were required to subtract the
number 13 from the number 467 for three con-
secutive repetitions. This simple task was shown
by Wickens et al. (1981) to be e?ective in clearing
the contents of working memory. In addition, the
demographic questionnaire itself acted as a dis-
tracter.
The test phase was designed to measure schema
acquisition. Schematic knowledge can be assessed
by having subjects (1) group problems into clus-
ters, (2) categorize problems after hearing only
part of the text, (3) solve problems when material
in the text is ambiguous, (4) identify information
in problems that is necessary and sucient for
solution, or (5) solve problems analogous to prac-
tice problems (Low & Over, 1992). This research
used the solution of analogous problems, due to
its rigor and ®t with our tax task. Solving the tax
problems requires both declarative knowledge of
the tax rules and procedural knowledge of their
application organized into problem-type schema
(Low & Over).
Completely unaided (i.e. no decision aid, no
instructions, and no notes) subjects were required
to solve ®ve problems that were analogous to the
practice problems and one transfer problem that
required rules covered in the practice problem
instructions, but not applied in a practice pro-
blem. The problems are de®ned as follows (a
sample problem appears in the Appendix).
1. A problem requiring knowledge of the con-
cepts of adjusted gross income, deductions,
and exemptions.
2. A problem requiring knowledge of the tax
liability computation rules for long-term
capital gains when taxpayers are in the 15 or
28% tax brackets (long-term capital gains
taxed as ordinary income).
3. A problem requiring knowledge of the tax
liability computation rules for long-term
capital gains when taxpayers are in tax
brackets above 28% (long-term capital gains
taxed at maximum marginal rate of 28%).
4. A problem requiring knowledge of the tax
computation rules for taxable social security
bene®ts when provisional income does not
exceed the ®rst base amount.
5. A problem requiring knowledge of the tax
computation rules for taxable social security
bene®ts when provisional income exceeds the
®rst base amount.
6. A problem requiring knowledge of the tax
computation rules for taxable social security
bene®ts when provisional income exceeds the
second base amount (transfer problem).
The six problems are arranged in a linear order
from simple to complex. To measure complexity,
we use the number of rules required for problem
solution (Low & Over, 1992; Sweller, 1988): pro-
blem one requires three rules, problem two
requires ®ve rules, problem three requires six rules,
problem four requires eight rules, problem ®ve
requires 11 rules, and problem six requires 12
rules. Problem one is required knowledge for all
other problems, but the remaining problems are
independent of one another with respect to
required knowledge (Low & Over). To control for
potential order e?ects in the test phase of the
J.M. Rose, C.J. Wolfe / Accounting, Organizations and Society 25 (2000) 285±306 293
experiment, the problems were randomly
ordered.
5
Rationale for the ordering of our test
phase problems rests on a ®nding by Low and
Over (1992) that indicates subjects fail to acquire
more complex schemata prior to the acquisition of
less complex schemata. To con®rm their results in
the tax domain, our six increasingly complex tax
problems were scored as either correct or incor-
rect. Fig. 2 shows the number of problems
answered correctly and incorrectly for all treat-
ments.
6
From Fig. 2, it is apparent that subjects
acquire this experiment's problem-type schema in
a linear fashion. That is, subjects who fail to
acquire simple schemata also fail to acquire more
complex schemata. The tree diagram in Fig. 2 can
be analyzed more formally using Guttman scalo-
gram analysis. The coecient of reproducibility
produced by Guttman scalogram analysis indi-
cates the predictability of the pattern of problem
solution. The coecient of reproducibility for the
six test phase problems is 0.989,
7
verifying that the
relative order of mastery in this group of tax pro-
blems is constant across subjects. This con®rmation
of Low and Over's ®nding is important, because it
indicates that subjects in this experiment with
higher scores on the test phase problems can solve
more complex tax problems than subjects with
lower scores, and likewise have acquired more
complex schemata.
5
All decision aids calculated the taxable social security ben-
e®ts before calculating tax on capital gains. This added strength
to the design because any recency e?ects would be in favor of
learning social security calculations before capital gains calcu-
lations. Schema theory predicts that capital gains schemata will
be acquired prior to social security schemata, because capital
gains schemata are less complex. Therefore, the acquisition of
capital gains schemata prior to social security schemata will not
be the result of a recency e?ect.
6
No scoring scale was used to produce Fig. 2. A problem
was scored as correct for this analysis only if there were no
errors in the solution.
7
The coecient of reproducibility was calculated using
Jackson's method (Maranell, 1974).
Fig. 2. Tree diagram of Test phase problems answered correctly.
294 J.M. Rose, C.J. Wolfe / Accounting, Organizations and Society 25 (2000) 285±306
2.4. Operational de®nitions of variables
2.4.1. Problem-solving performance (learning)
Problem-solving performance is measured by
the score obtained on the six problems in the test
phase of the experiment. The test phase problems
are completed without the use of a decision aid or
any tax calculation instructions after completing
the practice problems and a distracter task. The
test phase problems are scored 0±4 based on
accuracy and level of completion.
8
The total score,
which can range from 0 to 24, is used as a dependent
variable to represent problem-solving performance.
Di?erences in problem-solving performance represent
di?erences in the level of knowledge acquired,
because higher scores correspond to the successful
solution of increasingly complex problems.
2.4.2. Instruction integration (cognitive load)
The degree of instruction integration is varied
by di?erent placements of the tax calculation
instructions within each of the three decision aids.
The decision aid with instructions on a separate
screen has the lowest level of integration. The
decision aid with instructions on the same screen
has a moderate level of integration. The decision
aid with integrated instructions (i.e. instructions
adjoining the decision aid queries) has the greatest
level of integration.
2.4.3. Learning e?ort duration
Learning e?ort duration is captured by the time
spent working the three practice problems. The
time required to learn how to operate the decision
aid is removed from this measure by training sub-
jects on a decision aid devoid of any tax informa-
tion or instructions before they worked the
practice problems. All three decision aids can be
used to calculate tax liabilities in the same amount
of time. Di?erences in time spent working the
practice problems relate to di?erences in e?ort
duration directed towards learning the aid's
underlying calculation logic.
2.4.4. Problem-solving e?ectiveness
Problem-solving e?ectiveness is measured by the
number of correctly answered GRE questions.
There are eight GRE questions. Therefore, the
variable ranges from 0 to 8.
2.4.5. Problem-solving eciency
Problem-solving eciency is calculated based on
the time subjects spent solving correctly answered
GRE questions. The time required for each of the
eight questions varies and, therefore, a standar-
dized score for eciency is constructed. Each
subject's eciency is calculated as follows:
?Æ?TGE
ij
À MTGRE
i
?=SDTGRE
i
? Ã GRE
ij
??=
GRETOTAL
i
where, TGE
ij
=time spent on GRE question i for
subject j, MTGRE
i
=mean time spent by all sub-
jects on GRE question i, SDTGRE
i
=standard
deviation of time spent on GRE question i,
GRE
ij
=score (0,1) received by subject i on GRE
question j, GRETOTAL
i
=total score on GRE
questions (0±8) for subject i.
This formula yields an eciency score based on
the average deviation from mean solution times
for each subject. Lower score values indicate
greater problem-solving eciency.
9
3. Results
3.1. Preliminary statistics
A total of 287 subjects participated in the
experiment. Thirty-four subjects were removed
8
There was no need to compute the interrater reliability as
the grading scale was completely objective. The grader was
unaware of the treatments associated with each problem set.
9
Only times on GRE problems answered correctly were
used to calculate problem-solving eciency, because the inclu-
sion of problems answered incorrectly could contaminate the
eciency measure. However, it is also possible that subjects
who skip or gloss-over all problems could receive very high
eciency scores using our eciency metric. Therefore, two
alternative measures of problem-solving eciency that included
GRE questions not answered correctly were calculated. The
®rst alternative eciency measure was a similar standardized
score, but the score included times on all eight GRE questions.
The second and simplest measure was calculated by dividing
the total time spent solving GRE questions by the total number
of questions. Reported results do not change substantively
using either alternative eciency measure.
J.M. Rose, C.J. Wolfe / Accounting, Organizations and Society 25 (2000) 285±306 295
from the analyses because of prior knowledge or
failure to complete all aspects of the experiment.
10
Therefore, all analyses were conducted using the
remaining 253 subjects. The average age and GPA
of the subjects were 21 and 3.48, respectively.
Descriptive statistics and correlations are pre-
sented in Table 1.
3.2. Instruction integration
Hypothesis one proposes that subjects using deci-
sion aids with lower degrees of instruction integration
will have lower problem-solving performance (i.e.
increased cognitive load will reduce learning). This is
tested with an analysis of covariance (ANCOVA)
model using the instruction integration treatments
as a dependent variable, and the covariates, learning
e?ort duration, problem-solving e?ectiveness, and
problem-solving eciency, act as controls on the
independent variable, problem-solving performance.
As shown in Table 2, the ANCOVA model is statis-
tically signi®cant at the 0.0001 level.
11
Two of the
covariates, learning e?ort duration and problem-
solving e?ectiveness, are statistically signi®cant at
the 0.0001 level, and the treatment variable, degree
of instruction integration, is signi®cant at the
0.0001 level.
12
The Student±Newman±Keuls (SNK)
test for di?erences between least square means
indicates that after controlling for learning e?ort
duration and problem-solving e?ectiveness and
eciency, subjects in the integrated instructions
treatment learn more than subjects in the same
screen or separate screen treatments. These results
provide strong support for hypothesis one.
13
Table 1
Preliminary statistics
a
Variable N Mean Deviation Minimum Maximum
Panel A: descriptive statistics
Problem-solving e?ectiveness 253.00 5.95 1.23 2.00 8.00
Problem-solving eciency 253.00 À0.0032 0.55 À0.96 3.97
Problem-solving performance score 253.00 10.4200 5.55 0.00 24.00
Learning e?ort duration 253.00 856.62 464.46 329.00 3003.00
Problem-solving
performance
score
Problem-solving
e?ectiveness
Problem-solving
eciency
Learning
e?ort
duration
Panel B: correlation matrix
Problem-solving performance score 1.000 0.251
Ã
À0.080 0.454
Ã
Problem-solving e?ectiveness 1.000 À0.206
Ã
0.011
Problem-solving eciency 1.000 0.129
Learning e?ort duration 1.000
a
*Signi®cant at 0.05 level.
10
Any subject who had existing knowledge of the tax mate-
rial examined in our experiment was removed from the ana-
lyses. Prior knowledge was measured with a cued recall pretest.
11
Levene's test for homogeneity of variances indicates that
error variances were not equal across treatments. ANOVA,
however, is robust for violations of the homogeneity of var-
iance assumption when sample sizes are approximately equal.
To validate the results using a non-parametric procedure that
does not rely on a homogeneity of variance assumption, the
Kruskal±Wallis procedure and mean comparisons were con-
ducted using rank scores. Results for statistically signi®cant
di?erences were unchanged.
12
The analyses of treatment di?erences were repeated using
the ordinal ranking of the most complex problem solved as the
dependent variable. This measure of problem-solving perfor-
mance does not rely on any form of objective scoring. No
qualitative di?erences in results were found.
13
All analyses were repeated without including the score for
problem six, because problem six was not analogous to the
practice problems. No di?erences in results were found.
296 J.M. Rose, C.J. Wolfe / Accounting, Organizations and Society 25 (2000) 285±306
An important result stems from the no-aid
treatment group that performed all calculations by
hand. Consistent with prior research (Glover et
al., 1997; Murphy, 1990), the hand calculation
group's problem-solving performance was higher
than any decision aid treatment group. However,
when comparing the hand calculation group to
subjects using the integrated instructions decision
aid (i.e. the aid that produced the least cognitive
load), we found that the hand calculation group
had problem-solving performance scores that were
22% higher, but at cost of a 112% increase in
learning e?ort duration. When individual di?er-
ences in learning e?ort duration and problem-solving
e?ectiveness were controlled, problem-solving
performance di?erences were insigni®cant between
subjects doing hand calculation and subjects using
the integrated instructions decision aid. This ®nding
indicates that individuals of equal ability putting
forth equivalent e?ort can learn-by-doing equally
well, regardless of whether the process involves a
decision aid or not; as long as the decision aid
does not produce a large amount of cognitive
load.
3.3. Learning e?ort duration
The second hypothesis proposes that learning
e?ort duration will decrease as the degree of
instruction integration in the decision aid decrea-
ses (i.e. heightened cognitive load will reduce
learning e?ort duration). Table 3 displays
ANCOVA results and SNK least square mean
comparisons for learning e?ort duration while
controlling for problem-solving e?ectiveness and
eciency. Overall, the no aid treatment had higher
learning e?ort duration than any other treatment,
but no statistically signi®cant di?erence is found
between the decision aid treatments. The SNK
mean separation procedure does not support the
second hypothesis. Decreases in instruction inte-
gration did not lead to decreases in learning e?ort
duration. Given that results support the null for
hypothesis two, drawing any strong conclusion is
not feasible.
The third hypothesis states that decision aid
users with higher problem-solving eciency will
expend less learning e?ort duration. The
ANCOVA model in Table 3 indicates that pro-
blem-solving eciency is statistically signi®cant at
Table 2.
Analysis of problem-solving performance
ANCOVA and least square mean comparisons
a
Source Sum of square df Mean square F Signi®cance
Problem-solving e?ectiveness (covariate) 520.398 1 520.398 26.796 0.000
Learning e?ort duration (covariate) 273.001 1 273.001 14.057 0.000
Problem-solving eciency (covariate) 40.134 1 40.134 2.067 0.152
Between groups 856.126 3 285.375 14.694 0.000
Within groups 4777.531 246 19.421
Total 252
Di?erences in least square treatment means
Treatment LS mean Separate Same Integrated No aid
Separate screen 7.31 ± 2.69
Ã
4.97
**
4.98
**
Same Screen 9.99 ± 2.29
Ã
2.30
Ã
Integrated 12.28 ± 0.01
No aid 12.29 ±
a
R-square=0.385.
Ã
Signi®cant at p