Effectiveness of Instruction
I. Assessing the Effectiveness of Instruction at VCSU
Effectiveness of instruction at VCSU emanates from the educational goals reflected in the mission of the
university and the university’s deployment of the eight abilities.






Valley City State University is a learner centered community.
The University is dedicated to continuing improvement in meeting student needs.
The University is preparing individuals to serve in a changing world.
The University is a leader in the effective use of instructional technologies.
The University provides a quality educational experience in an innovative culture
Student’s comprehension of the eight University Abilities is fundamental to the curriculum.
Goal
Valley City State University is a
learner centered community.
Measurements
 Faculty post office hours
 Course evaluations
 Syllabi are readily available to students
The University is dedicated to
 Faculty attend and present at local and national conferences.
continuing improvement in
 Program reviews are used to strengthen programs. (See Method Below)
meeting student needs.
 Faculty attend campus workshops and trainings
 Faculty embraces new and innovative methods of offering courses.
(blackboard course offerings)
The University is preparing
 The General Education Block provides a basis for future learning.
individuals to serve in a changing  Employer surveys
world.
 CD-ROM Portfolios
The University is a leader in the  Campus research on Technology use on the campus
effective use of instructional
 Faculty focus group research
technologies.1
 Technology training are regularly available on the campus
(Results Section IV)
 Syllabi Research
The University provides a quality  Faculty are knowledgeable in their fields (vitas)
educational experience in an
 Adjunct faculty are competent in their fields (vitas & hiring policies).
innovative culture.
 Course offerings are based on national standards specific to academic
curriculum.
 Student surveys “Learner’s Survey”
 Graduate surveys
 The campus has obtained Faculty development grants that focus on
current instructional methods. (Bush, PT3, Title III, Title III 98)
Student’s comprehension of the
 The Abilities are fundamental to the Gen Ed. Experience
eight University Abilities is
 All programs are involved in providing project-based learning with the
fundamental to the curriculum.
abilities.
Defined by Armsey & Dahl (1973) as the “things of learning”, the devices and materials which are used in the process of learning
and teaching
1
II. Instructional Effectiveness at the Department/Divisional Level
Instructional effectiveness is monitored primarily at the department/division level. However, the process by
which departments/divisions ensure that instruction is delivered effectively is subjected to periodic review.
(WHAT IS THAT PERIOD?)
 How often?
 Initiated by who?
 Who reviews a committee or the VPAA?
 After review what happens to results, how are they used, where are they recorded?
Recommended Activities
Program Review plan.
Student Assessment of teaching
and learning
Student assessment of courses
in program
Student Assessment plan
(leading to CD-ROM)
Gen Ed. “outcomes” identified
that provide benchmarks for
incoming majors.
CD-ROM: comparison of
entrance and exit materials
Map abilities & technology for
program
Faculty and peer assessment of
courses (external standards)
Faculty development plan to
stay current in field
Faculty portfolios
Faculty involvement with
students to improve learning
climate.
Senior Portfolio
Best work website
Graduates exit survey
Alumni Survey
Examples of Assessment Methods
See VCSU policy manual, 403.1.2; plan is due in VPAA office Nov. 21.
Program improvement loops should be indicated.
 Flashlight project (see Terry Corwin) allows assessment of specific
aspects or general course evaluation. Items are keyed to Seven
Principles of Good Practice in Undergrad Ed.
 Use current form with additional questions; protect confidentiality of
information about faculty.
 Use external standards (NCATE, state, national organizations)
 Possibly combine with above student assessment.
Plan should show entrance expectations, process of student movement
through program, and expectations at graduation, including portfolio
assessment.
This may be part of Student assessment plan.
Both entrance and exit materials COULD be on the same CD. Program
should compare the two sets of material to evaluate effectiveness of
program.
Could be in tabular (matrix) form.



Use external standards (NCATE, state, national organizations)
Use student evaluation of courses and faculty’s professional judgment.
Faculty may have a development plan.




Portfolios are required for promotion and tenure
Non-tenured and tenured faculty are assessed on a regular basis
Advising
Student club (program) sponsorship
Can be used a part of exit assessment of students in major.
TBA
TBA--we are attempting to coordinate this campus-wide
TBA—we are attempting to coordinate this campus-wide
Survey of Employers
External review
TBA—Marcia Walker will coordinate
Coordinate with VPAA—not all programs may use this option.
III. Research Background on VCSU’s Assessment Model for Teaching & Learning
This brief review of the relationship between technology and education begins with John Dewey. He believed
that technology should serve the process of inquiry. In viewing Dewey as the first great philosopher of
technology, he regarded inquiry as a productive craft, and technology as the tools of the craft (Dewey, 1938). In
more recent times, some educators embraced new technologies such as audio, video and computer aided
instruction. Researchers began to tell us that technology made no difference in student learning. While
researcher continued to tell us that the majority of teachers were not using, let alone integrating technology into
their teaching, one researcher took a different view of the situation. Clark argued, “The Medium is not the
message. Communication media and other technologies are so flexible that they do not dictate methods of
teaching and learning. All the benefits attributed by previous research to "computers" or "video" could be
explained by the teaching methods they supported. Research should focus on specific teaching-learning
methods, not on questions of media” (Clark, 1989).
This new focus for research, while it has merit, also has its difficulties from the researcher’s viewpoint. It means
integrating technology into the methods of teaching and learning, not laying the technology over the top and
then removing it from a control group. Many factors in the educational classroom cannot be controlled, nor in
most cases would researchers attempt to try. Even the definition of a classroom has changed with the latest Web
technologies. Educators like Ehrmann (1995) of the American Association of Higher Education suggest we
approach the complex problem in this manner:
1.
Technology often enables important changes in curriculum, even when it has no curricular
content itself.
2.
What matters most are the educational strategies for using technology, strategies that can
influence the student's total course of study.
3.
If such strategies emerge from independent choices made by faculty members and students,
the cumulative effect can be significant and yet still remain invisible.
To assess changes in learning a university must study its educational strategies for using
technologies. It is not possible to measure these strategies in a single course but it must be done across
the institution or division if the evolution of the strategies is to be monitored. (pg. 25)
Other educators like Gilbert support this perspective and suggest it is necessary to have a density of technology
use before changes in learning can be appropriately measured. Gilbert (1996) states the following, “To make
visible improvements in learning outcomes using technology, use that technology to enable large-scale changes
in the methods and resources of learning. That usually requires hardware and software that faculty and students
use repeatedly, with increasing sophistication and power. Single pieces of software, used for only a few hours
are unlikely to have much effect on graduates, lives or the cost-effectiveness of education.”
Based on the research and writings of the above authorities on technology and learning, VCSU has
created and implemented a research model.2 The results of which will assist in assessing changes in teaching
and learning on the campus. The major segments of the model are as follows:
1.
2
ACT and high school GPA scores of incoming freshmen are recorded as indicators of overall
change in the student population.3
A flowchart of the model is included
2.
3.
4.
5.
6.
VCSU is a technology rich learning environment, not just because of the laptop environment
that surrounds our students, but previous campus research reveals our faculty have
significantly increased their technology use every year since the initiative began and that 79%
now indicate computer technology is essential to their teaching. Faculty technology use is
followed on a regular basis through a survey, syllabi review and faculty focus groups.2
The Senior Portfolio on CD-Rom4 is a digital review of what students have learned while at
VCSU. The portfolios are based on projects completed in courses throughout the curriculum.
Each project is constructed around the University’s Abilities and Skills. The multimedia
capabilities of the computer make it possible for students to demonstrate their understanding
of the Abilities. In 2002 every VCSU graduate will complete a digital portfolio. VCSU is
currently one of only undergraduate institutions in the nation to require digital portfolios of all
of its graduates.
Assessment of VCSU’s student perceptions of the learning on the campus has been followed
since 1996. A survey created by a faculty member, with constructs based on the Seven
Principles of Good Practice in Undergraduate Education (Chickering & Gamson, 1987 has
been administered every year.5
Graduate data needed for the outcomes portion of the plan is gathered yearly through the
VCSU Placement Office. Surveys have been recently modified to meet the needs of the plan.2
Program assessment is also a key part of the assessment process and is the one of the feedback
factors used in campus wide model. Continued improvement of this process is an important
aspect in understanding the needs of the student and the appropriate inclusion of course
learning objectives.
This assessment model was begun as a method of assessing the changes on the campus created by a Title III
grant. The grant is responsible for the implementation of the Senior Portfolios and will end in September of
2000. The assessment process was expanded and continues to be modified to better register changes in teaching
and learning across the campus. The model was modified using suggestions from Ehrmann’s 1997 article How
(Not) to Evaluate a Grant-Funded Technology Project. He states:
Considering a grant-funded project as part of a larger pattern of change implies two quiet different
evaluative tasks – finding out:
1. Is that larger pattern of improvement having a good effect on learning outcomes for graduates? Or
are there other reasons to think that it will, aside from direct evidence (e.g., research in other
institutions that shows that this teaching and learning strategy usually has good outcomes)?
2. If so, is your innovation playing a useful role in the maintenance or growth of that larger change in
strategy? Or, on the other hand, is your innovation neutral or even interfering with that improvement
in overall teaching and learning strategy? (Pg. 5)
A timeline for completing the data gathering for each of the segments in the model in included in this report.
More classroom assessment activities and survey results that furnish faculty with a better understanding of how
the “Seven Principles” are affecting student learning are two areas that need further study. Continuing to
improve the model, facilitate faculty research, capture and analyze data, and document the results is a major
undertaking.
3
See attached graphs indicating results of data
More information about the portfolios is available on the Website http://www.vcsu.nodak.edu/offices/TitleIII
5
Results of this year’s survey and general trends are available in Appendix A.
4
Several projects underway on the campus position the faculty for further improvements in the assessment
process. Two of these projects are promoted by the writings of Gilbert and Ehrmann. These authors are
supporters of the Flashlight Project and the Teaching and Learning with Technology Roundtable (TLTR). Their
writings emphasis the importance of applying the teaching strategies outlined in Seven Principles of Good
Practice in Undergraduate Education (Chickering & Gamson, 1987). VCSU has purchase the Flashlight
Current Student Inventory and is sending a faculty member to Flashlight’s upcoming workshop. VCSU also
plans to implement a TLTR experience with its faculty.
IV. Research on Technology and Effectiveness of Instruction: Documenting Faculty’s Use of Technology
Quantitative Data – Benchmarking faculty’s use of the available technology is the first step in the process.
Five quantitative indicators and one qualitative indicator were used to determine technology adoption. The four
quantitative indicators in Figure 6 show statistically significant increases in each of the first four years since
1996. In the fifth year, three of the indicators have slight but not significant declines.
 Faculty indicated how many of the available technologies they used
in preparation or presentation of their teaching.
# of Tech Uses by Faculty
8
6.33
6
4
7.15
6.63
4.37
3.11
2
0
1996 1997 1998 1999 2000
Year
 Faculty indicated how many of the available technologies they
require students to use in their courses.
# of Tech Uses by Students
6
4.83
5.71
3.67
4
2
5.22
2.06
0
1996 1997 1998 1999 2000
Year
 Faculty indicated how frequently they used the software available
on their notebook computers and in the Title III multimedia lab.
Frequency of Software Use by Faculty
37.41 36.03
40
24.94 27.19 27.18
30
20
10
0
1996 1997 1998 1999 2000
Year
 Faculty indicated their level of computer use (LCU) based on the
Marcinkiewicz (1993) questionnaire.
Level of Use
8
6
5.81
6.51
6.7
7.3
6.91
4
2
0
1996 1997 1998 1999 2000
Year
Figure 6: Four Quantitative Indicators of Faculty Technology Use
The fifth quantitative indicator of technology use was a random sample of 30 courses selected from the 300 to
400 taught on the campus during a one-year period. See Figure 7 for a graphic representation.
In the first set of 30 course syllabi collected for the 1995-96 academic year, only 7 syllabi or 23% indicated
computer technology use by the students or faculty. There were 8 total uses in the thirty courses. Only one
course had more than one use (2) and three of the syllabi had just the instructor’s E-mail address as a use.
(Figure 8)
However, in the second set of 30 syllabi (1996-97), 24 or 80% had at least one indication of technology use
by students or faculty. There were 51 total indications in the second set of syllabi. Of the single uses only two
were simply an E-mail address. Two syllabi had five uses, two had four uses and thirteen or 43% had more than
one. Fifty-seven percent more of the syllabi had at least one technology indication in the 1997 than in 1996, and
there were 43 more total indication in the group overall, an increase of 84% over the one year period. (Figure 8)
In the third set of 30 syllabi (1997-98), 26 or 86% had at least one indication of technology use by students
or faculty. This is an increase of 6%. There were 49 total indications in the third set of syllabi. Of the 10
single uses only five were simply an E-mail address. One syllabus had 4 uses, four had 3 uses and 17 or 57%
had more than one. (Figure 8)
In the forth set of 30 syllabi (1999-00), 28 or 93% had at leas on indication of technology use by students or
faculty. This is a 7% increase from the previous sample. There were 72 total indications in the forth set of
syllabi. Only 4 syllabi had only one indication and those were E-mail addresses. Three syllabi had 5 uses, two
had 4 uses and 24 or 80% had more than one. These results of the four samples show a continued stable and
increasing application of technology in VCSU courses since the 1996-97 school year. (Figures 7 & 8) The
complete results of technology application are available in the 2000 Summative Title III Report.
100%
Percent of Syllabi with Technology Included
93%
80%
Percent
60%
66%
60%
40%
23%
20%
0%
1995-96
1996-97
1997-98
Year
1999-00
Figure 7: Indicates Yearly Increase in Technology Application in Random
Syllabi Sample
Number of Syllabi
Technology References from 30 Syllabi
35
30
25
20
15
10
5
0
14
16
24
Two or More Uses
1
3
3
1996
9
2
1997
4
5
1998
4
4
2000
One Computer Use
(any)
One Use: Email
Address Only
Figure 8: Two Graphs Showing Various Changes in Syllabi Technology
Inclusions
Qualitative Data – Over the past five years three faculty focus groups have been conducted on the campus. For
each a random sample of 5-7 faculty were selected. The focus groups provide a more in-depth perspective of the
campus culture. Focus groups were organized following the survey research profiled above and questions were
asked to further clarify the research data.
Focus Group 1997 The focus group was conducted in December of 1996 and April of 1997 one semester
following the distribution of notebook computers to students on VCSU campus. Eight faculty members attended
the two one hour focus groups. They represented the divisions/departments of Health and Physical Education,
Education, Mathematics, Business and Communication Arts.
The responses to the questions were almost totally positive. It would be difficult to identify any type of overall
negative reflection. The negatives were usually expressed as complains about the need for more technology
(networked classrooms or better student computers). Some concerns were over the loss of the network or lack of
it at one location on the campus. Also there was concern that some faculty did not use computers enough in
their teaching. However, the complains were tempered with assumptions that it would improvement. Even when
specifically asked about pressure and negative concerns, the faculty responded as follows:
 “Administration have an influence on us – can be a positive influence but is a type of pressure.”
 “Influenced -- Yes with no pressure. Students expect us to use computers so the instructor needs to be
ready.”
 “To be innovated and stay innovated…”
The one theme that was repeated overwhelmingly through out the session was student and student centered. In
fact, the terms “student” was repeated fifty times in the transcripts of the focus group responses. The faculty at
VCSU seem to be very concerned about the needs of their students with reference to technology. Statements
such as those that follow were easy to fine in the interview transcripts:
 “The instructor and the student are learning together while the instructor is the facilitator.”
 “Students have more pride in their work.”
 “There is much more accessibility to information through the Internet. The instructor and the students
are both more comfortable using computers and showing their computer skills.”
There seemed to be a change taking place in the way teaching was happening on the campus. The faculty seem
to want to continue and promote the change.
Focus Group 1998 The focus group was conducted in March of 1998 a year and a half following the
distribution of notebook computers to students on the VCSU campus. Eight faculty attended the one hour focus
group. They represented the divisions/departments of Health and Physical Education, Education, Mathematics,
Business, Science, and Computer Instructional Systems.
The responses to the questions were almost totally positive. Even when asked about concerns over the
changes on the campus most of the concerns were architectural matters such as classroom design, lighting and
difficulties with off campus classrooms. One instructor stated, “The Jamestown classroom is like going back to
cave days.” The major theme of the discussion was students and student learning. Comments such those below
were common:

“The computers helps students get organized”

“Notebooks computers enhance our ability to motivate the students to be better learners.”

“There are increases in creative writing and it is (the computer) a great organizer for writing papers”
Spring 1998 was the fourth semester since the infusion of notebook computes and other technologies into the
campus environment. Many of the focus group comments reflected the wisdom gained from trial and error or;
they now know more about what works, how well and what doesn’t work. They agreed that e-mail enhanced
communication with students but that it works best for surface types of management situations. Face to face
discussions are still valued. Faculty enjoy increased communication with colleagues and alumni through e-mail
and listservs. Receiving and grading material in digital form does not work for everyone. Hard copy is still the
preferred way for many to grade written papers. Comments concerning the use of hard copy included: it is
easier to write in comments, it takes less time to pick up the paper and read it, hard copy can be read anywhere
and enough time is spent on the computer doing other activities.
A second related theme was the changes in the student population, both in incoming freshmen and in the
current students. Faculty agreed technology made it easier to meet the needs of more students. After three
semesters with technology this is some of the growth and change noted by the faculty:

Students have become less resistant to using technology and group projects are easier to manage.

Hands on training with course specific software has made learning of important concepts easier and less
time is needed for simply learning to use software.

Using the web to enhance learning through simulation sites and immediate access to information was
very important to instruction but control of student computer activity during class was also important.
One faculty reflects that, “In networked classroom you do not know if the student is taking notes,
surfing the web or playing games.”

Technology has increased hands on experiences, decreased lecture time and resulted in less “spoon
feeding” of students.

Member of one department noted there had been no change in their teaching since the infusion of
campus technology except for its distraction in the classroom.
Generally, the faculty seem pleased by what they are accomplishing and they are comfortable with
technology and the changes happening on the campus.
Focus Group 2000
I NEED TO WRITE THIS
V. A framework for Assessing (Academic) Achievement:
A. VCSU faculty apply technology as a tool in their instructional methods.
See Section III & IV of this document.
B. VCSU has identified, implemented and assessed a set of instructional outcomes (Abilities & Skills)
throughout the General Education and Major Curriculums.
See Section VI of this document.
C. Student surveys indicate VCSU faculty apply instructional methods that utilize both AAHE’s Principles
for Good Practice in Undergraduate Education and appropriate technology.
The following documents the constructs of the Learner Survey constructed and distributed by Dr.
Kathryn Holleque
The Learner Survey Constructs
In the beginning, the American Psychological Association and Mid-continent Regional Educational Laboratory
identification of learner centered psychological principles was used as a basis for The Learner Survey
constructs. Use of The Seven Principles by Chickering and Gamson, (1987) and adoption of the Flashlight
Project followed.
The “Seven Principles of Good Practice in Undergraduate Education” developed by Chickering and Gamson,
1986; Chickering and Ehrmann, 19966 include:
1.
2.
3.
4.
5.
6.
7.
Interaction between the student and teacher
Student-student interaction
Active learning
Time on task
Rich, rapid feedback
High expectations of the student’s ability to learn
Respect for different talents, way of learning
These seven principles were identified in answer to the question, “What does research indicate about the
conditions that produce good learning?” According to the wider body of research on teaching and learning upon
which Chickering and Gamson drew, when these principles are implemented, learning outcomes usually
improved, too.
Questions in The Learner Survey are coded according to the fourteen factors listed below related to technology
use and created through the Flashlight Project2 (VCSU has purchased the rights to utilize this material). A
notable overlap exists between these factors and the Seven Principles. Thus, if the Seven Principles and
technology use are applied to teaching, the university can track changes through The Learner Survey.
By grouping the questions by the code and tracking change over time we can look at increases or decreases in
these factors across the university. (We are in the process of double coding with the Seven Principles, as well.)
A
C
=
=
D
E
=
=
F
G
I
N
=
=
=
=
O
P
S
=
=
=
T
U
X
=
=
=
6
2
Active learning
Collaborative learning (and other forms of student-student
interaction
Using time productively
High expectations for all students regardless of learning
style
Rich and rapid feedback
Engagement in learning
Faculty-student interaction
Cognitive and creative outcomes (including encouraging
creativity)
Accessibility
Positive addiction to technology
Prerequisites for using technology (technical skill
deficiencies)
Time on task
Respect for diversity
Application to “real world” problems/preparation for work
This essay can be found on the world Wide Web at http://www.aahe.org/technolgy/ehrmann,htm
URL for the Flashlight Project: http://www.wiche.edu/flshlght/flash.htm
Code
OP
X
#
4.
5.
C
I
UN
C
GE
6.
7.
8.
9.
10.
DC
NG
UC
X
AG
XA
FA
E
E
E
D
O
P
GN
E
IE
N
11.
12.
13.
14.
15.
16.
17.
18.
19.
20.
21.
22.
23.
24.
25.
26.
27.
IC
FA
DN
P
PG
P
F
F
F
U
28.
29.
30.
31.
32.
33.
34.
35.
36.
37.
Question
Is it important to me to have my own computer to use.
The various technology skills I am developing at this university are essential to my future
employment.
Using a computer increases my communication with other students.
Using a computer increases my communication with faculty.
Having my own computer broadens ways for me to receive and/or present information.
This university fosters cooperation in learning.
Using a computer provides a variety of setting for learning (e.g.. working by myself, forking
with others, working as a member of a team).
Using technology tools makes it easier to work in groups.
Using technology tools increases my critical thinking.
I feel a diminished sense of competition in learning at this institution.
This university provides real-life learning opportunities.
Having my own computer makes it easier to be actively involved in the learning process.
I am involved in projects that promote student choice and responsibility.
I use the Internet often to research topics for courses.
Having my own computer helps me assume personal responsibility for learning.
I am empowered to meet my individual learning needs at this university.
Using a computer makes it easier to meet my learning needs.
Having my own computer saves me time.
It is important to me to have computer access at any time, day or night.
Using a computer is a positive addiction for me.
Using a computer supports my curiosity and intrinsic interest in learning.
Using a computer allows me to experience success in meeting my learning goals.
This university has a supportive, positive learning environment.
Using a computer enables me to integrate and organize knowledge in personally meaningful
ways.
The learning environment at this university is non-threatening.
I use the Internet regularly to find academic information.
Using a computer helps me pursue meaningful goals.
I enjoy taking courses that make use of Internet materials.
Using various technologies enhances my learning experience.
Using a computer during class is valuable to my learning.
I access the Internet daily.
I use my computer daily.
I use e-mail daily.
The learning environment at this university broadens my appreciation for the diversity
among people.
VI. Documenting Ability-Based Student Assessment
The second group of benchmarks was developed from a set of Abilities and Skills unique to VCSU. The
Abilities were established and endorsed by the faculty and the Skills, while originally based on the SCANS
Skills,7 were modified to incorporate the university’s perspective. Both were adopted by the Faculty Senate in
the Spring 2000 and listed in the 2000-2002 Bulletin. A booklet (Abilities – Skills – Levels) is available.
DIVISIONAL, GENERAL EDUCATION AND THE MONSTER MAPS ARE DOCUMENTATION
NEEDED HERE.
VII. The Institutional Improvement Committee







7
Spring of 1999 the Institutional Improvement Committee (IIC) was charged by the president Chaffee as
the Steering committee for the 2001 NCA Self Study.
Members of the committee represent each of the six academic divisions, students and the community, as
well as the VPs of Student Affairs, Business Affairs and Academic Affairs.
In the fall of 1999 the committee review the Self Study Plan completed by the Co-coordinators. It was
submitted to NCA in the Fall of 1999.
For organizational purposes, after reviewing the Plan’s Timeline, the committee members self-selected
themselves to chair each of the five criterion committees and a sixth partnership and entities committee.
Two committee members co-chaired each of the committees. Co-chairs were charged with finding other
members for their committee and creating subcommittees where necessary
During Fall Semester, subcommittees examine the GIRs relating to their Criterion or area of study and created
reports that were given to the Criterion co-chairs.
The IIC met regularly throughout the Fall semester to discuss problems and report on the work of the
subcommittees
Spring of 2000 the results of the GIRs committee work was collated and turned in for review by the cocoordinators.
Secretary’s (of Labor) commission on acquiring necessary skills
References
Armsey, J.W., & Dahl, N.C. (1973) An inquiry into the uses of instructional technology. New York: Ford
Foundation Report p. vii
Chickering, A. & Gamson, Z. (1987). Seven principles of good practice in undergraduate education,” AAHE
Bulletin (March).
Clark, R. (1998). Current progress and future directions for research in instructional technology. Educational
Technology Research and Development 37(1), 57-66.
Dewey, J. (1938). Logic: The theory of inquiry. New York: Henry Holt.
Ehrmann, S. C., (1995). Asking the right questions: What does research tell us about technology and higher
learning? Change. XXVII: 2 (March/April), pp. 20-27.
Ehrmann, S. C., (1997). How (not) to evaluate a grant-funded technology project. Flashlight Evaluation
Handbook (Washington DC:TLT Group 1997). Website http://www.tltgroup.org/flashlight/evaluate.html
Gilbert, S. W. (1996). How to think about how to learn. AGB Trusteeship. Special Issue 1996.