Collaborative Evaluation of Learning Objects

 

 

John C. Nesbit

Karen Belfer

Simon Fraser University

 

 

 

Abstract. The properties that distinguish learning objects (LOs) from other forms of educational software – global accessibility, metadata standards, finer granularity and reusability – have implications for evaluation. This chapter proposes a collaborative model for LO evaluation in which representatives from stakeholder groups (e.g., students, instructors, subject matter experts, instructional designers, and media developers) converge toward more similar descriptions and ratings through a two-stage process supported by online tools. The chapter reviews evaluation models that have been applied to educational software and media, considers models for gathering and meta-evaluating individual user reviews that have recently emerged on the Web, and analyzes the peer review model adopted for the MERLOT repository. The proposed convergent participation model is compared to other models and assessed with respect to its support for eight goals of LO evaluation: (1) aid for searching and selecting, (2) guidance for use, (3) formative evaluation, (4) influence on design practices, (5) professional development and student learning, (6) community building, (7) social recognition, and (8) economic exchange.

 

 

 

This chapter is based on work published in:

 

Nesbit, J. C., Belfer, K., & Vargo, J. (2002). A convergent participation model for evaluation of LOs. Canadian Journal of Learning and Technology, 28 (3), 105-120.

When instructors and students search a repository to select a learning object (LO) for use, they have three questions in mind: Is it the right type? Is it the best I can find? and How should I use it? Substantial effort has been spent in recent years building metadata standards and tools to assist users in answering the first question. Now, with repositories filling with thousands of objects, many dealing with similar subject matter and learning goals, greater attention and effort will turn to the second and third questions.

LO evaluation by third-party reviewers is a key element in promoting reusability because the availability of quality ratings can have an immediate and compelling effect on the outcome of repository searches. Quality ratings are already being used to order search results in MERLOT, a leading LO metadata repository for higher education. In MERLOT, highly rated LOs are returned ahead of objects that have lower ratings or have not been evaluated. The use of quality ratings in this way places significant responsibility on the e-learning research community to develop demonstrably valid evaluation methods because even minor variations in the method may have amplified effects on many thousands of users’ selection decisions.

The challenge of developing effective evaluation systems is formidable because they must optimize on two opposing variables: the number of objects that can be evaluated versus the quality of the obtained evaluations. A costly model that returns highly accurate evaluations is of little use to the individual user if it can only be applied to a small fraction of a collection. Also, it is insufficient for reviews to return only numeric quality ratings: Reviews can most strongly benefit users by elaborating on pre-existing metadata to describe the situations and contexts in which the objects can be appropriately used.

Some definitions of LOs admit non-digital resources and cover a wide range of aggregation levels including whole courses and certificate programs (IEEE, 2002). But, because our goal is to facilitate reusability through quality evaluation, this chapter is primarily concerned with digital resources at the lower to middle levels of granularity; a scope that includes static images or text, animations, simulations, interactive lessons, tests, and so on.

We have identified eight interrelated reasons for developing effective LO evaluation systems:

  1. Ratings and qualitative assessments aid individual users in searching and selecting objects.
  2. Evaluations can provide guidance on how best to use an object.
  3. Quality can be increased by formative evaluation throughout the design and development stages.
  4. Evaluation standards can drive the practices of designers and developers.
  5. Participation in evaluation activities can contribute to the professional development of those who work with LOs.
  6. Evaluation activities can build and support communities of practice in relation to LOs.
  7. Positive evaluations can promote social recognition of skilled designers and developers.
  8. A trusted evaluation system is an essential step toward the development of a workable business model for the economic exchange of LOs.

Here we propose a convergent participation model that addresses all of the above goals.  In the proposed model evaluation is conducted by a panel drawn from different stakeholder groups. For example, in a college setting, learners, instructors, instructional designers, and media developers might participate as panel members.  There is a two-cycle process. During the first cycle the evaluators independently and asynchronously assess the object. During the second cycle, which may be conducted synchronously or asynchronously, the evaluators compare and explain their earlier assessments, adjusting their individual assessments as their judgment shifts in response to the panel’s discussion.

During both cycles the evaluators use an assessment instrument that gathers numeric ratings and comments on a small number of dimensions. The process is managed by a moderator who guides discussion but does not directly assess the object. During the second cycle the moderator sequences the items under discussion to ensure that features about which there is greatest disagreement are discussed first.

In the convergent participation model evaluators use online tools that enable fully distributed participation. The tools support the communication of individual assessments among group members and aggregation of individual assessments into collaborative reviews.

 

Evaluation of Educational Software and Multimedia

In considering some of the alternative approaches to evaluating educational software and multimedia, we use the classification scheme of Worthen, Sanders and Fitzpatrick (1997) who distinguish between, among others, consumer-oriented, expertise-oriented, objectives-oriented, and participant-oriented evaluations.

Consumer-oriented evaluations of educational materials are conducted by governmental organizations and non-profit associations that train evaluators, often teachers, to apply standard criteria, checklists or rating scales, to examine the materials and produce reviews in a highly structured format. With costs in the same range as those for the production of basic LO metadata, consumer-oriented evaluation is one of the least expensive models, and for this reason alone is highly applicable to the problem of assessing large numbers of LOs. Containing over 7000 reviews of K-12 materials, the EvaluTech repository (Southern Regional Education Board) is perhaps the best evidence of the cost-efficiency of a consumer-oriented model. The quantitative instruments often used in consumer-oriented evaluation provide a score for each object that enables repositories to order search results. Also, structured reporting formats have the advantage of facilitating comparison among objects.

The validity and descriptive power of consumer-oriented evaluations are limited by the same factors that make them cost-efficient. Evaluators typically work individually, unable to benefit from the specialized expertise of others. The depth of analysis, and the flexibility to deal with the special characteristics of objects are restricted to whatever can be built into the instrument and procedure on which the evaluator is trained. In reviewing instruments for educational software evaluation, Gibbs et al. (2001) noted that the instruments have been subject to criticism for not being comprehensive, understandable, and easy to use. There is evidence of low correlations among evaluators within the criteria that they use for evaluations, usually content, interface design and technical operation  (Jolicoeur & Berger, 1986).

Expertise-oriented evaluations are conducted by recognized experts, either individually or in panels. In comparison with consumer-oriented approaches, they place less emphasis on structured formats, closed-response instruments, and training; relying instead on the expert judgment of the evaluator. The MERLOT peer review process, described later, can be regarded as combining the expertise and consumer-oriented approaches because it brings expert evaluators together with standard scoring criteria and structured reports.

Expertise-oriented approaches have been criticized as being especially vulnerable to the subjective biases of the evaluators. And, they often show low inter-evaluator consistency because individual expert evaluators tend to place greater importance on the specific factors that form the basis of their own expertise (McDougall & Squires, 1995; Reiser & Dick, 1990). As in the consumer-oriented model, there is no representation of stakeholders such as learners, and no testing of the object in situ. There may be representation from only a single expert community – perhaps only subject matter experts or only instructional design experts. Expertise-oriented approaches tend to be more costly than consumer-oriented approaches because they lack the efficiency advantages of a fixed procedure repeatedly applied by a trained evaluator.

Objectives-oriented approaches couple detailed analysis and definition of goals with empirical, quantitative studies using pre-post or comparative designs that test the extent to which the goals have been attained. Reiser and Dick (1990) developed a model for educational software evaluation that involved, in part:

Similar models have been used for demonstrating the efficacy of software for teaching skills such as spelling and fractions (Jolicoeur & Berger, 1988; Assink & van der Linden, 1993).  A major drawback of objectives-oriented approaches is the cost associated with running an empirical study. Given our current systems, it is far more expensive to treat and test a group of students than to have a trained evaluator fill out a form. Another limitation of this approach is that in emphasizing goals and outcomes, objectives-oriented approaches tend to ignore the learning processes that occur as students interact with materials.

Unlike the consumer-, expertise-, and objectives-oriented approaches, participant-oriented approaches to educational evaluation explicitly acknowledge that learning is a social process dependent on social context. This implies that multimedia and software evaluation must account for interactions between the learner and those around them (Baumgartner & Payr, 1996). The methodology of participant-oriented evaluation is drawn from naturalistic and ethnographic research: Data is gathered by prolonged and persistent observation, informal interviews, and document analysis; and reported as detailed descriptions or direct quotations (Neuman, 1989; Patton, 1980). There is often an emphasis on bringing stakeholders together in moderated discussion groups:

 

Participants in the group process become sensitized to the multiple perspectives that exist around any program. They are exposed to divergent views, multiple possibilities, and competing values. Their view is broadened, and they are exposed to the varying agendas of people with different stakes in the evaluation. This increases the possibility of conducting an evaluation that is responsive to different needs, interests, and values (Patton 1982, p. 65)

 

The recognition that program evaluation reports are often underutilized or ignored has led to an interest in promoting organizational learning through the participant group process. Working within an activity theory framework in a university environment, Dobson and McCracken (2001) credited the success of an evaluation project that improved teaching and learning to a team-based approach in which academics and learning technology experts worked as equal partners in furthering innovation. Although the analysis is outside the scope of this chapter, activity theory (Nardi, 1996) offers a relevant perspective on all the evaluation approaches we describe. With its depiction of collective activity (e.g., collaborative evaluation) operating on an object to produce an outcome (e.g., a published review) through the use of mediating tools (e.g., an online evaluation instrument), it is especially consistent with the convergent participation model we are proposing.

Williams (2000) proposed two participant-oriented models that an organization can use to evaluate the LOs it creates. One model deals with externally contracted evaluations of larger instructional units that include LOs. It involves competitive requests for proposals, stakeholder interviews, and the possible use of the delphi technique to develop key questions to be addressed by the evaluation. This is followed by the formulation and execution of an evaluation plan. The second model deals with formative, internally conducted evaluations. It builds an evaluative component into every step of the ADDIE instructional design model (Assess needs, Design, Develop, Implement and Evaluate instruction). Both the internal and external models require meta-evaluation as a final stage.

There are features of most participant-oriented approaches that are incompatible with the requirements of LO evaluation. Often these approaches tailor the evaluation’s guiding questions to situational demands to obtain an evaluative report with a unique format. This practice would make comparison of large numbers of LOs difficult. In participant-oriented approaches there are usually no quantitative ratings that can be used to sort search results, even though this is an extremely convenient feature that many users have already come to expect. Finally, extensive collection and analysis of detailed qualitative data is just too costly for use with large numbers of objects.

In the search for an efficient model that preserves many of the advantages of participant-oriented approaches, we were impressed with many of the interactive tools for communication and collaboration provided in online communities. By providing automated functions that facilitate voluntary contributions from users, these websites demonstrate the strengths of technology-mediated communities managed by their members.

 

Lessons from the Web

User ratings and comments have become much easier to gather with the establishment of the Internet. Increasingly sophisticated systems are emerging that address the problem of poor quality contributions by classifying, filtering, and sorting user comments on the basis of meta-evaluative ratings provided by other users. Featuring brief articles that may each be discussed by hundreds of comments posted by users, the Slashdot website (www.slashdot.org), led this trend by allowing certain community members to rate others’ comments so that any reader could view only comments whose average rating fell above an adjustable threshold. By authoring highly rated comments, a member can acquire the prestigious “karma points” that increase one’s likelihood of being temporarily granted the privilege of rating others’ comments. The whole system is extremely effective in ratcheting up the quality of user comments and bringing the most valued comments into the foreground.

The evaluation requirements of a LO repository are similar in several ways to those of an online book retailer. Amazon’s user review and recommendation system (www.amazon.com) offers significant insights into how object repositories might manage user reviews. Any Amazon user can submit a 1000-word book review including a rating on a 5-point scale. Other users can vote yes or no on the usefulness of the review, generating an approval rating that determines the order in which reviews are displayed. The user ratings of a book are combined with a customer’s buying preferences and expressed interests to construct a personalized list of recommended books that are presented to the customer when entering the site.

Websites presenting consumer-authored reviews of items ranging from cameras to travel destinations further illustrate the design and operation of online evaluation communities. With over nine million unique visitors per month, and over a million published reviews, the Epinions website (www.epinions.com) is perhaps the best example of a venue in which producing and accessing reviews, not purchasing products, is the central transaction. Users can register in Epinions and create personal profiles including a photo, biography, link to homepage, favorite websites, and links to reviews they have authored. Epinions supports meta-evaluation of both the review and reviewer. Reviews can be rated by any member on a 5-point scale ranging from Very Helpful to Not Helpful. Of greater significance for community building, though, is the "web of trust" linking member profiles. Members can choose to trust or block the authors of reviews they have read. Links to profiles of trusted others are presented on the member's own profile, forming a navigable network of shared interest and trust relationships. Epinions members can be promoted to Top Reviewer if they (a) frequently write reviews that consistently receive high ratings and (b) maintain trust links to other authors of other highly rated reviews. Members can be promoted to Advisor if they frequently contribute high quality meta-evaluations. Aside from social recognition for high quality contributions, the Top Reviewer and Advisor roles relate to the visibility and sort order of published reviews: Reviews authored by Top Reviewers, or highly rated by Advisors, are more prominently displayed.

Do Internet users pay attention to trust and reputation metrics? A study by Resnick and Zeckhauser (2002) on trust and reputation in the online auctions site eBay (www.ebay.com) found both a high rate of transaction evaluation (> 52%) and a low rate of negative or neutral evaluations (1%). Items were more likely to be sold when the seller had a high reputation. When sellers were given a negative rating they often (29% of the time) posted an explanation in an attempt to avoid reputation damage. Especially in review websites, trust and reputation take on differing functions that go beyond the role of predicting honest behaviour. Trust can indicate interpersonally shared interests and attitudes. The "web of trust" is really a sub-network of members with similar beliefs and goals. Reputation, on the other hand, tends to signal community-wide norms and standards. A "Top Reviewer" models the attitudes and review style valued by the majority, and thereby carries the ability to shift community values in new directions.

Noticeably lacking from existing review websites is the opportunity for dialogue among reviewers and other users, or for collaborative reviews. For reasons we present in this chapter, we believe that, in the case of LO evaluation, promoting relevant discussion and engaging members in review panels will lead to higher quality reviews and stronger incentive for community participation.

The individual user reviews made possible by the Internet offer huge cost advantages over the more formal evaluation approaches we have discussed; and they seem essential for any comprehensive system of LO evaluation. But the effectiveness of meta-evaluation mechanisms like those that control quality of user contributions on Slashdot, Amazon, and Epinions has only been demonstrated on high traffic sites, and for objects that attract many reviewers and meta-evaluators. LO review sites, that we presume would serve a smaller market, might need higher levels of community participation to benefit from these same meta-evaluation mechanisms. Perhaps a key to designing whole LO evaluation systems is to find ways for user reviews and formal reviews to inter-operate in a complementary fashion. For example, a member who completed a sufficient number of highly rated individual reviews might be invited to participate in formal panel reviews. Formal panel reviews might resolve cases where an object received strongly conflicting individual user reviews. Formal reviews might also be expected to set standards emulated by individual reviewers.

 

MERLOT

In contrast to other large metadata repositories, MERLOT <www.merlot.org> has established a set of relatively mature evaluation practices and tools. Emphasizing disciplinary expertise, the approach to LO evaluation in MERLOT is largely modeled on the academic peer review process for scholarly research and publication familiar to university faculty. The MERLOT web site currently supports 14 discipline specific communities, each with an editorial board that guides peer review policies and practices.

Anyone can self-enroll to become a member of MERLOT and submit individual member comments on materials registered with the repository. Peer Reviews, on the other hand, are conducted by members of a discipline community, usually two university faculty with relevant content expertise. After the reviewers prepare ratings and comments individually, one of them combines the individual assessments and averages the ratings to create an integrated review that may be edited by the other reviewer and discipline co-leaders.

The discipline communities usually select higher quality objects for peer review. Of the 8157 objects listed in MERLOT at the time of this writing, 22% had at least one member comment and 9% had a peer review, For both member comments and peer reviews, a large majority of evaluated objects were rated at level 4 or 5. We conjecture that large sampling bias in favor of higher quality objects will be found in almost all systems of LO evaluation. Although effort might be invested in culling obsolete or low quality objects on the basis of prima facie judgment, a practice that has been adopted within some MERLOT discipline communities, there is little benefit from allocating significant resources to evaluating low quality items. The MERLOT peer review system was designed to encourage adoption within the academic culture of university faculty. The MERLOT organization’s careful attention to the values of that culture, such as professional volunteerism and respect for disciplinary knowledge, seem to have worked well in sustaining growth of the collection, the membership, and the discipline communities. However, in our view, the MERLOT peer review process needs to be extended to match the conditions under which LOs are developed and used, and to better meet the needs of students. Unlike journal articles written by and for academic researchers, high quality LOs are very often created by teams that include faculty, instructional designers, media developers or programmers; and they are used by instructors and students. As we have discussed earlier, assessment models that include representative participation from all stakeholder groups are more likely to result in valid evaluations that will be used by practitioners.

Repositories like MERLOT that emphasize community building and evaluation features would do well to implement meta-evaluation and trust systems similar to those we described earlier in this chapter. We believe that filtering out lower quality reviews and highlighting reviews that receive top ratings increases members' motivation to submit thoughtful reviews and boosts the value of the service for all users. In addition to governance structures such as editorial boards, true communities are rooted in the interpersonal relationships of their members. Community websites can strengthen these relationships by explicitly representing them as hyperlink networks connecting members with shared interests and beliefs.

 

Convergent Participation

The convergent participation model, depicted in Figure 1, is a panel evaluation process designed to obtain better outcomes than the peer review model without resorting to expensive field studies. When applied to the evaluation of LOs, it is presumed to exist in an online community, similar to MERLOT, in which users post individual member reviews that include comments and ratings. Panel reviews are organized by a moderator who selects one or more objects for review according to criteria established by the community. The moderator also selects and invites panel participants that represent different stakeholder groups, some of whom may have already posted individual member reviews of the object(s) to be evaluated by the panel.

In both cycles of this two-cycle model, the reviewers use an evaluation instrument capable of gathering ratings and comments specific to several different features of LOs. In the first cycle, through an asynchronous process lasting several days, the participants examine the object and submit an individual review no different from those they might post outside the panel evaluation process. Those who had already submitted a review might edit their existing review or simply revisit it to recall the reasoning behind their judgment.

In the second cycle, the reviews from all participants are exposed in an integrated format, and the moderator leads a discussion focusing on the points of greatest divergence among the participants. The moderator’s role is to keep the discussion on track without revealing views he or she may have about the object or otherwise biasing the judgment of the participants. A critical characteristic of the model is that the reviews produced in the first cycle are used by the moderator to sequence the discourse so that


Figure 1. Convergent Participation with LORI

 

features of the LO about which there is least agreement are discussed first. As the second cycle proceeds the participants may edit their individual reviews using tools that immediately update the integrated view available to the entire panel. When the time allocated for the second cycle expires, the moderator brings the discussion to a close and asks all participants to approve the publication of the panel review consisting of the integrated ratings and comments. Integrated reviews show the range and central tendency of individual ratings with comments concatenated within different evaluative categories. Only data from participants who approve publication are included in published review.

Several features of the model have been left undetermined: some because they seem situation dependent, and others because we have insufficient experience with the model to make a well-founded recommendation. Our preference is to conduct the second cycle as a real-time meeting supported by synchronous communication tools, but there may be conditions under which asynchronous discussion would be more suitable. Anonymity of the participants, both among themselves and in relation to the larger community, is a potentially important element that invites further investigation.

In a recent study we tested a prototype of the convergent participation model with 12 participants divided into three internationally distributed panels (Vargo, Nesbit, Belfer & Archambault, 2003). The participants included instructional designers, university faculty and media developers. Each participant evaluated eight LOs: four objects were individually evaluated and four were evaluated through convergent participation. The participants used an early version of the Learning Object Rating Instrument (LORI). LORI presents a 5-point scale for each of the nine items shown in Table 1 (Belfer, Nesbit, & Leacock, 2003). Inter-rater reliability was measured by separate intraclass correlations (Shrout & Fleiss, 1979) for first-cycle and second cycle evaluations. The results showed more consistent convergence (i.e., increased inter-rater reliability) for the collaboratively evaluated objects than the individually evaluated objects. Some participants spontaneously commented that the process was an excellent way to develop their knowledge about LOs. Participant comments also established that training on some of the dimensions of LORI is a requirement for effective use of the instrument.

 

Table 1. Items in LORI (version 1.4)

 

There are points of similarity between the Convergent Participation model and the practice of focus groups as described by Kreuger (1994). As with focus groups, the moderator’s role is to facilitate a conversation among the participants, not to conduct a series of interviews with individual participants. In addition to pacing the discussion and maintaining focus, the moderator must establish an environment where participants feel comfortable to express differing views and support their views through argumentation.

Although there is evidence that participants, in both the proposed model and in focus groups (Kreuger, 1994), do converge in their views as a result of discussion, the emergence of consensus is not a required or even desired outcome of either approach. In using the convergent participation model, one hopes that participants will come to a common understanding of the instrument and how it applies to the LO, but it is expected that reviews will often be published that show disagreement among the reviewers.

One might question why the whole panel is expected to judge dimensions on which only some participants have professional expertise. Although in principle participants can select a “don’t know” option, the moderator does encourage all participants to contribute a rating for each dimension. This practice invites any expert on the dimension under consideration to explain the rationale behind his or her ratings and comments so that others are persuaded to rate in a similar way. When two experts disagree, the other ratings will sway to the side that can most persuasively communicate its argument to the panel.

 

Support for the Goals of LO Evaluation

In the introduction to this article we offered the claim that the convergent participation model addresses eight goals for LO evaluation. We are now in a position to discuss the extent of support it provides for each of these goals, relative to support provided by other models.

Aid for searching and selecting. Like the peer review model, the convergent participation model is likely too costly to fully cover a large repository containing many thousands of LOs. We believe that it can complement individual user reviews and consumer-oriented reviews by offering better quality evaluations in high demand areas and for objects that for any reason require greater attention and analysis. We hope that, by drawing a broader cross-section of participants into the collaborative review process, a larger base of trained reviewers can be established to carry out high quality individual reviews. The model seems well suited to comparative reviews that evaluate a few objects occupying overlapping curricular space, although it has yet to be extended or tested for that application.

Guidance for use. The extent to which a convergent participation review suggests ideas to instructors and learners about how best to use an object depends on the nature of the instrument that structures the discussion. We would argue, though, that a discussion on lesson planning or assignment design that includes instructional designers, learners, subject matter experts and instructors will offer greater insight than one in which only one of these roles is represented.

Formative evaluation. Convergent participation reviews can certainly reveal strengths and weaknesses in an early version of an object in a format that would be very useful for a development team. On the other hand, in its current form the model does not offer the continuous monitoring that is regarded as optimal for formative evaluation. The model could be extended to work within an instructional design or project planning model, perhaps by substituting different instruments as appropriate at different project milestones. We caution that, as an evaluation model, convergent participation cannot substitute for the range of functions carried out by an instructional designer.

Influence on design practices. Regardless of whether one is considering curricular, instructional, or media design, an evaluation standard will tend to drive design practices if it represents the values of the community of designers and developers. When one community determines the standards used to evaluate the work of other communities, tensions arise between communities, evaluations are underutilized, and individual design practices are unaffected. Convergent participation is a boundary-crossing activity that should assist in establishing common evaluation standards that are accepted across represented communities and are capable of influencing individual practice within each community.

Professional development and student learning. When communicating across the boundaries of disciplinary expertise it is often necessary to expose tacit assumptions and press implicit knowledge into explicit language. In our view this has great educative value for all concerned and forms ideal conditions for professional development. Further, if the instrument adopted with the model is derived from research, the participants are naturally led into an increased familiarity and critical understanding of the underlying research base. Under some conditions, it seems justifiable to assign course credit or professional development release time for work on convergent participation review panels.

Community building. The work of evaluating LOs in an intensely collaborative setting both expends, and creates in greater measure, what Putnam (2000) calls social capital, the pro-social motive normally generated in any matrix of active interpersonal relationships. Social capital is community-building glue that can drive community members to act in the common interest, perhaps contributing unpaid hours to observe learners interacting with a new user interface, or spending a portion of sabbatical time on re-designing a curriculum. Because the activity of developing high quality LOs requires the cooperation of different professional groups, it is important that social capital generated in the evaluation activity be spread across those groups to form an inclusive community.

Social recognition. Although many different evaluation systems can inform the granting of professional awards, these surely carry greater social recognition value when established in a social context wider than the specialized professional guild to which the recipient belongs. A media development award will garner more respect among professors when professors are represented in the evaluative process that granted it.

Evaluation for economic exchange. The quality of evaluative information is very often a deciding factor in the purchase of any commodity. The availability of higher quality evaluations may reduce licensing costs by eliminating the need for the purchaser to carry out in-house evaluations. We argue that evaluation from any single disciplinary perspective, whether that is content, instructional design, or media design, is insufficient; and further that those who buy LOs to assemble commercially offered courses are most interested in the views of the end-user, the learner.

To summarize, the strengths of the convergent participation model are that it brings together representatives of stakeholder groups, efficiently focuses their attention on the points that may be in greatest need of resolution, and produces a review that concisely presents areas of agreement and dissent among the evaluators. Advancement of the model requires more study of how converging participants interact, along with the development of tools to support that interaction.

 

References

 

Assink, E., & van der Linden, J. (1993). Computer controlled spelling instruction: A case study in courseware design. Journal of Educational Computing Research, 9(1), 17-28.

Baumgartner, P., & Payr, S. (1996). Learning as action: A social science approach to the evaluation of interactive media. Conference on Educational Multimedia and Hypermedia. Association for the Advancement of Computing in Education. Retrieved April 27, 2002, from http://www.webcom.com/journal/baumgart.html

Belfer, K., Nesbit, J. C., & Leacock, T. (2003). LO evaluation instrument (LORI), version 1.4. Unpublished manual.

Dobson, M., & McCracken, J. (2001). Evaluating technology-supported teaching and learning: A catalyst to organizational change. Interactive Learning Environments, 9(2), 143-170.

Gibbs, W., Graves, P. R., & Bernas, R. S. (2001). Evaluation guidelines for multimedia courseware. Journal of Research on Technology in Education, 34(1), 2-17.

IEEE Learning Technology Standards Committee (2002). LO Metadata Standard IEEE 1484.12.1. Retrieved September 7, 2002, from http://ltsc.ieee.org/wg12/

Jolicoeur, K., & Berger, D. E. (1986). Do we really know what makes educational software effective? A call for empirical research on effectiveness. Educational Technology, 26(12), 7-11.

Jolicoeur, K., & Berger, D. E. (1988). Implementing educational software and evaluating its academic effectiveness: Part I. Educational Technology, 28(9), 7-13.

Krueger, R. A. (1994). Focus groups: A practical guide for applied research (2nd edition). London: Sage.

McDougall, A., & Squires, D. (1995). A critical examination of the checklist approach in software selection. Journal of Educational Computing Research, 12(3), 263-274.

MERLOT. Evaluation standards for learning materials in MERLOT. Retrieved February 9, 2003, from http://www.merlot.org/eval.html.

Nardi, B. (1996). Context and consciousness: Activity theory and human-computer interaction. Cambridge, MA: MIT Press.

Neuman, D. (1989). Naturalistic inquiry and computer-based instruction: Rationale, procedures, and potential. Educational Technology, Research & Development, 37(3), 39-51.

Patton, M. Q. (1980). Qualitative evaluation methods. Beverly Hills: Sage.

Patton, M. Q. (1982). Practical evaluation. Beverly Hills: Sage.

Putnam, R. (2000). Bowling alone: The collapse and revival of American community [Electronic version]. New York: Simon & Schuster.

Reiser, R. A., & Dick, W. (1990). Evaluating instructional software. Educational Technology, Research and Development, 38 (3), 43-50.

Resnick, P., & Zeckhauser, R. (2002). Trust among strangers in Internet transactions: Empirical analysis of eBay's reputation system. The economics of the Internet and e-commerce. In M. R. Baye (Ed.). Advances in applied microeconomics (vol. 11). Amsterdam: Elsevier Science.

Shrout, P. E., & Fleiss, J. L. (1979). Intraclass correlations. Psychological Bulletin, 86, 420-428.

Southern Regional Education Board. EvaluTech. Retrieved April 24, 2002, from http://www.evalutech.sreb.org/

Vargo, J., Nesbit, J., Belfer, K., & Archambault, A. (2003). LO evaluation: Computer mediated collaboration and inter-rater reliability. International Journal of Computers and Application, 25 (5).

Williams, D. D. (2000). Evaluation of LOs and instruction using LOs. In D. A. Wiley (Ed.), The instructional use of LOs [Electronic version]. Retrieved April 18, 2002 from http://reusability.org/read/chapters/williams.doc

Worthen, B. R., Sanders, J. R., & Fitzpatrick, J. L. (1997). Program evaluation: Alternative approaches and practical guidelines (2nd ed.). New York: Longman.