Important Characteristics of Course Management Systems:
Findings from the Edutools.info project

Scott Leslie, Researcher, Edutools.info project

(Note: This paper was presented at the 2003 CADE Conference, June 8-11, St. John's Newfoundland)

Abstract

Since they first emerged in the mid-1990s, Course Management Systems have evolved from somewhat basic collections of disparate applications into intricate, integrated enterprise systems. Through the work of the Edutools project, this paper looks at the features that most commonly define course management systems, and using site usage data collected from the decision making tool, whether these features are in fact what the users of course management tools are seeking.

Project Background

The EduTools website (http://www.edutools.info/course/) was launched in September 2002 by the Western Cooperative on Educational Telecommunications (WCET). It is a web-based database containing independent reviews of 45 Course Management Systems (CMS). Its purpose is to allow users interested in CMS to compare up to 10 products side-by-side on up to 41 features. The site also contains an idealized decision making tool. This tool allows users to select a number of features on which to compare products, assign importance weightings for each of the features to be compared, and also to assign ‘suitability scores’ (user assessments on how well a product fulfils a particular feature.) The ‘decision’ is then produced by multiplying the feature’s weighting and its suitability score.

The site is actually a redevelopment of work originated by Dr. Bruce Landon and the Centre for Curriculum, Transfer in Technology from 1996 to 2001 (http://www.c2t2.ca/landonline/, no longer functioning) on a site entitled “Online Educational Delivery Applications: A Web Tool for Comparisons”.

As part of the site re-development, extensive tracking features were built in the hopes that a picture would emerge of how the site was being used. A somewhat incidental by-product of this usage tracking is that the research team now had access to data on which products were most often being compared, what features were most commonly supported across products, and which features were most used as the basis for comparison and decisions. This paper will present the data collected since September 2002 and report on whether the features most often supported by CMS are the same features most often used as the basis of decisions between CMS. It will also report on those cases where supported features are not being used as often in decision making, as well as differences in feature use between product comparison and product decision making.

Definition of Course Management Systems

For the purpose of the Edutools site, CMS were defined partly in reference to their functionality and partly in reference to their current usage and interest within our target audience, decision makers in the higher education community. One of our primary concerns was to differentiate Course Management Systems from

• Learning Management Systems (typically aimed at corporate training and functional more focused on learner tracking and competency management)
• Learning content Management Systems (again most often found in corporate settings and typical used to create and manage learning content)
• Content Management Settings (typically focused solely on the creation and maintenance of generic web site content with no pedagogical tools or insight) and,
• products aimed at the K-12 market, which share some aspects with CMS but need to address additional audiences (e.g. parents) and interoperate with different back-end systems.

Higher-Education Target Market defined

In order to assess whether a specific product had higher education as its target market, the following factors were considered:

• the software is being provided from within higher education
• the product is primarily used in higher education
• the software company’s website explicitly describes higher education as a target market
• existing customers include higher education institutions
• the product integrates with academic-oriented back office systems (e.g. student record systems such as SCT Banner or Datatel's Colleague) versus more industry-oriented ERP or HR systems (e.g. SAP or BAAN)
• the scope of content partners includes traditional academic publishers versus more industry-oriented workplace learning publishers.

CMS Functional Definition

From a functional perspective, we defined CMS partly by features they should include, and partly by what they should not be or include. The working definition of course management systems excludes:

• Single function software like stand-alone assessment tools, synchronous tools or authoring packages that do not also have many other features or act as part of a larger suite that delivers online education courses.
• Course content materials and course content bundled with its own online delivery system.

The working definition of course management systems includes:

• Software suites that enable both synchronous and asynchronous student participation with both the learning content and with other students and the instructor.
• Systems sold in a traditional vendor model that allows institutions to run the software in their own environment and hosted services provide through Application Service Provider (ASP) style relationships.
• Software packages and hosted services when they provide at least one of the following functions:
  • Tools to facilitate multiple aspects of course design, content authoring, and content management.
  • Tools for administering assessments and tracking the usage from both a student and content perspectives.
  • Tools that structure content delivery and course progression around conventional post-secondary course units of delivery and post-secondary schedules such as terms and semesters.

The Edutools ‘feature set’

The feature set on which Edutools researches and reviews CMS has evolved since being originally defined in 1997 through a consultative process with an advisory group comprised of post-secondary educators and educational technology specialists. Dr. Bruce Landon then modified this initial set of features to accommodate new features as they developed across this gradually maturing domain of systems. (The full site history can be seen at http://www.edutools.info/about/history.jsp as well as the members of the initial project team and advisory board.)

The new site took the evolved set of features as its starting point, but also undertook a review process with a new advisory board (http://www.edutools.info/about/advisory.jsp).

In using the initial feature set to review a growing number of products, the researchers have begun to get a clearer picture on the breadth (and sometimes lack) of support for certain features. Additionally, as people have used the decision making tool, we have gathered statistics on which features people consider more important in making their decisions. Both of these metrics have helped us to further refine the feature set and consolidate some of the features into cohesive groupings. [N. B. As a result, a total of 14 of the original 55 features were either deprecated or consolidated into other features in January 2003. Efforts have been made to reflect this when possible, but because these edits reflected lack of prevalence or usage typically these features did not impact the top 30 features in feature prevalence or decision usage.]

The full set of features along with both their Basic and Technical definitions can be found at http://www.edutools.info/course/help/glossary.jsp. Below is a table listing the features by name, presented in the same format as they are in the site, grouped in conceptual categories by intended user group (Learners, Instructors, Technical Administrators) and broad function.

Table 1 – Edutools Feature Set

Bold indicates Conceptual Categories that are used for organizational and display purposes only)

Findings

Common Features of CMS

As a result of reviewing 45 CMS on 41 features, the researchers at Edutools were able to form a picture of some of the common and not so common features of these systems. Table 1 shows the top 15 features supported across these systems. Of the 45 systems reviewed, Angel 5.5 and WebCT Vista were found to support the highest number of features (38) and Edusystem the lowest (9), with the average number of features supported being 29, or about 70% of the total feature set.

We found it notable that there is not one individual feature that can be said to define course management systems. That said, there are a number of features that appear to be supported across most of the systems. Of note are Discussion Forums (41 products), Registration Integration (41), Internal Email (39), and Authentication (38).

Also notable were those features that were most often lacking. Features that were substantially below the average were Video Services (supported by only 7 products), Student Community Building (17), Whiteboard (16), Curriculum Management (10) and Open Source (10). The definition employed of ‘”Video Services” restricts this feature to only streaming media or the support of video conferencing, so it is somewhat unsurprising that these higher-end and specialized technologies are not supported generally. In the case of “Curriculum Management” this feature is in part a crossover feature to the sphere of Learning Management Systems, and again it is unsurprising that we did not find it commonly supported.

The full number of products supporting each feature can be found in the graph in Appendix 1. This analysis is based on data that is publicly available on the edutools site by tallying up the features on the ‘All-by-All’ comparison sheet at http://www.edutools.info/course/compare/all.jsp.

Table 2 – Top 15 Features Supported across Products

Features used in Decisions vs. Support amongst products

In addition to acting as an informational resource to users on the features of CMS, Edutools provides a tool through which users can make product decisions based on self-selected features. Table 3 lists the top 15 features used as criteria in these decisions.

Table 3 – Top 15 Features used in Decision Making Tool

On it’s own this information is of limited interest. Of greater interest is a comparison of the features used in decision-making with the numbers of products generally supporting each feature. In theory what emerges is a picture of the extent to which this particular software market segment is in tune with what its users are actually looking for.

The chart in Appendix 2 depicts both of these measures expressed as percentages of their totals. While it may be expected that 3 of the under-represented features of CMS (Video Services, Whiteboard and Curriculum Management) also show noticeable differences here, what seems more notable are the number of features supporting pedagogical content development and deployment that appear to be of importance to users in choosing products but which are not necessarily widely supported. These include ‘Course Management,’ ‘Instructional Design,’ and the aforementioned ‘Curriculum Management’ tools.

Conversely, features like ‘Internal Email,’ ‘Hosted Services, ‘Course Authentication’ and ‘Course Authorization’ are somewhat widely supported across products, and yet they appear to be used less often in the decision making tool. Rather than pointing to a feature that users do not want or find important, in the case of the latter two a reasonable explanation could also be that both Course Authorization and Authentication are such basic elements of a CMS as to be taken for granted.

Custom Criteria

In addition to using any of the existing 41 features on which Edutools reviews products as the basis for a decision, the decision tool allows users to enter their own custom criteria. Table 4 details the top ten criteria supplied independently by users when making decisions.

Seemingly reinforcing the above observation that users are seeking (and perhaps not always finding) instructional content development and deployment capabilities, 7 of these top 10 user-supplied criteria might reasonably be understood as pertaining to such functionality.

Table 4 – Top 10 Custom Criteria used in Decision Tool

Problems with this Study/Possible Areas for Improvements

Top of the list phenomenon

The list of features on which users could compare products and on which they could make decisions never varied in order, and within each of the conceptual categories the features were presented in alphabetical order. Thus users may have succumbed to the general tendency to select from the top of the list. It is possible that this contributed, for instance, in the prevalence of ‘Discussion Forums’ in both comparisons and decisions, though it should also be noted that in this case it is also the most widely supported feature.

Inclusion in decisions is not a full indication of ‘importance’

While the usage of a feature to make a decision does lend it some significance versus features that were not used at all, the Edutools site itself acknowledges in the design of its decision tool that not all features are to be considered of equal importance, and thus allows users to assign weightings in their decisions. Ideally these weightings would also be considered and factored against the feature support across products, but for a variety of reasons, including those of privacy, we have chosen not to store this data.

Different types of users focus on distinctly different sets of CMS features

The analysis presented in this paper would have benefited much from knowing what type of user (Instructor, Institutional Decision Maker, Technical Administrator, Educational Researcher, Student) was using both the comparison and decision tools, as we expect these different sorts of users would focus on quite distinct sets of features. The decision making tool is built specifically to accommodate this diversity of users in its accommodation of weightings. Again, for a variety of reasons, both those of privacy and ones of usability and user-adoption, a decision was made early on in the project to not collect this information.

Appendix 1 – Chart – Number of Products Supporting Features

Appendix 2 – Chart – Feature Support vs. Usage in Decisions