Important Characteristics of Course Management Systems:
Findings from the project

Scott Leslie, Researcher, project

(Note: This paper was presented at the 2003 CADE Conference, June 8-11, St. John's Newfoundland)


Since they first emerged in the mid-1990s, Course Management Systems have evolved from somewhat basic collections of disparate applications into intricate, integrated enterprise systems. Through the work of the Edutools project, this paper looks at the features that most commonly define course management systems, and using site usage data collected from the decision making tool, whether these features are in fact what the users of course management tools are seeking.

Project Background

The EduTools website ( was launched in September 2002 by the Western Cooperative on Educational Telecommunications (WCET). It is a web-based database containing independent reviews of 45 Course Management Systems (CMS). Its purpose is to allow users interested in CMS to compare up to 10 products side-by-side on up to 41 features. The site also contains an idealized decision making tool. This tool allows users to select a number of features on which to compare products, assign importance weightings for each of the features to be compared, and also to assign ‘suitability scores’ (user assessments on how well a product fulfils a particular feature.) The ‘decision’ is then produced by multiplying the feature’s weighting and its suitability score.


The site is actually a redevelopment of work originated by Dr. Bruce Landon and the Centre for Curriculum, Transfer in Technology from 1996 to 2001 (, no longer functioning) on a site entitled “Online Educational Delivery Applications: A Web Tool for Comparisons”.


As part of the site re-development, extensive tracking features were built in the hopes that a picture would emerge of how the site was being used. A somewhat incidental by-product of this usage tracking is that the research team now had access to data on which products were most often being compared, what features were most commonly supported across products, and which features were most used as the basis for comparison and decisions. This paper will present the data collected since September 2002 and report on whether the features most often supported by CMS are the same features most often used as the basis of decisions between CMS. It will also report on those cases where supported features are not being used as often in decision making, as well as differences in feature use between product comparison and product decision making.

Definition of Course Management Systems

For the purpose of the Edutools site, CMS were defined partly in reference to their functionality and partly in reference to their current usage and interest within our target audience, decision makers in the higher education community. One of our primary concerns was to differentiate Course Management Systems from

Higher-Education Target Market defined

In order to assess whether a specific product had higher education as its target market, the following factors were considered:

CMS Functional Definition

From a functional perspective, we defined CMS partly by features they should include, and partly by what they should not be or include. The working definition of course management systems excludes:

The working definition of course management systems includes:

The Edutools ‘feature set’

The feature set on which Edutools researches and reviews CMS has evolved since being originally defined in 1997 through a consultative process with an advisory group comprised of post-secondary educators and educational technology specialists. Dr. Bruce Landon then modified this initial set of features to accommodate new features as they developed across this gradually maturing domain of systems. (The full site history can be seen at as well as the members of the initial project team and advisory board.)

The new site took the evolved set of features as its starting point, but also undertook a review process with a new advisory board (

In using the initial feature set to review a growing number of products, the researchers have begun to get a clearer picture on the breadth (and sometimes lack) of support for certain features. Additionally, as people have used the decision making tool, we have gathered statistics on which features people consider more important in making their decisions. Both of these metrics have helped us to further refine the feature set and consolidate some of the features into cohesive groupings. [N. B. As a result, a total of 14 of the original 55 features were either deprecated or consolidated into other features in January 2003. Efforts have been made to reflect this when possible, but because these edits reflected lack of prevalence or usage typically these features did not impact the top 30 features in feature prevalence or decision usage.]

The full set of features along with both their Basic and Technical definitions can be found at Below is a table listing the features by name, presented in the same format as they are in the site, grouped in conceptual categories by intended user group (Learners, Instructors, Technical Administrators) and broad function.

Table 1 – Edutools Feature Set

Bold indicates Conceptual Categories that are used for organizational and display purposes only)

Learner Tools

Communication Tools

Discussion Forums

File Exchange

Internal Email

Online Journal/Notes

Real-time Chat

Video Services


Productivity Tools


Calendar/Progress Review


Searching Within Course

Work Offline/Synchronize

Student Involvement Tools



Student Community Building

Student Portfolios

Support Tools

Administration Tools


Course Authorization

Hosted Services

Registration Integration

Course Delivery Tools

Automated Testing and Scoring

Course Management

Instructor Helpdesk

Online Grading Tools

Student Tracking

Curriculum Design

Accessibility Compliance

Course Templates

Curriculum Management

Customized Look and Feel

Instructional Design Tools

Instructional Standards Compliance

Technical Specifications


Client Browser Required

Database Requirements

Server Software

Unix Server

Windows Server


Company Profile


Open Source

Optional Extras

Software Version



Common Features of CMS

As a result of reviewing 45 CMS on 41 features, the researchers at Edutools were able to form a picture of some of the common and not so common features of these systems. Table 1 shows the top 15 features supported across these systems. Of the 45 systems reviewed, Angel 5.5 and WebCT Vista were found to support the highest number of features (38) and Edusystem the lowest (9), with the average number of features supported being 29, or about 70% of the total feature set.

We found it notable that there is not one individual feature that can be said to define course management systems. That said, there are a number of features that appear to be supported across most of the systems. Of note are Discussion Forums (41 products), Registration Integration (41), Internal Email (39), and Authentication (38).

Also notable were those features that were most often lacking. Features that were substantially below the average were Video Services (supported by only 7 products), Student Community Building (17), Whiteboard (16), Curriculum Management (10) and Open Source (10). The definition employed of ‘”Video Services” restricts this feature to only streaming media or the support of video conferencing, so it is somewhat unsurprising that these higher-end and specialized technologies are not supported generally. In the case of “Curriculum Management” this feature is in part a crossover feature to the sphere of Learning Management Systems, and again it is unsurprising that we did not find it commonly supported.

The full number of products supporting each feature can be found in the graph in Appendix 1. This analysis is based on data that is publicly available on the edutools site by tallying up the features on the ‘All-by-All’ comparison sheet at

Table 2 – Top 15 Features Supported across Products

Feature Name

# Products Supporting Features

Percentage of Total (45) Products

  1. Discussion Forums



  1. Registration Integration



  1. Internal Email



  1. Authentication



  1. Real-time Chat



  1. Orientation/Help



  1. Groupwork



  1. Self-assessment



  1. Course Authorization



  1. Automated Testing and Scoring



  1. File Exchange



  1. Calendar/Progress Review



  1. Instructor Helpdesk



  1. Searching Within Course



  1. Student Tracking



Mean # of products supporting a feature 29, Standard Deviation 8.72

[N.B. The following features were removed from this list as they are not strictly speaking functional features and as such are present in nearly all products, the exceptions being where the information was not made available:Company Profile, Software Version, Optional Extras, Costs, Client Browser Requirements]

Features used in Decisions vs. Support amongst products

In addition to acting as an informational resource to users on the features of CMS, Edutools provides a tool through which users can make product decisions based on self-selected features. Table 3 lists the top 15 features used as criteria in these decisions.

Table 3 – Top 15 Features used in Decision Making Tool

Feature Name

Number of Times Feature Used in Decision

Percentage (out of 1720 total decisions)

  1. Discussion Forums



  1. Course Management



  1. File Exchange



  1. Student Tracking



  1. Automated Testing and Scoring



  1. Online Grading Tools



  1. Internal Email



  1. Groupwork



  1. Real-time Chat



  1. Self-assessment



  1. Course Templates



  1. Authentication



  1. Instructional Design Tools



  1. Calendar/Progress Review



  1. Customized Look and Feel



On it’s own this information is of limited interest. Of greater interest is a comparison of the features used in decision-making with the numbers of products generally supporting each feature. In theory what emerges is a picture of the extent to which this particular software market segment is in tune with what its users are actually looking for.

The chart in Appendix 2 depicts both of these measures expressed as percentages of their totals. While it may be expected that 3 of the under-represented features of CMS (Video Services, Whiteboard and Curriculum Management) also show noticeable differences here, what seems more notable are the number of features supporting pedagogical content development and deployment that appear to be of importance to users in choosing products but which are not necessarily widely supported. These include ‘Course Management,’ ‘Instructional Design,’ and the aforementioned ‘Curriculum Management’ tools.

Conversely, features like ‘Internal Email,’ ‘Hosted Services, ‘Course Authentication’ and ‘Course Authorization’ are somewhat widely supported across products, and yet they appear to be used less often in the decision making tool. Rather than pointing to a feature that users do not want or find important, in the case of the latter two a reasonable explanation could also be that both Course Authorization and Authentication are such basic elements of a CMS as to be taken for granted.

Custom Criteria

In addition to using any of the existing 41 features on which Edutools reviews products as the basis for a decision, the decision tool allows users to enter their own custom criteria. Table 4 details the top ten criteria supplied independently by users when making decisions.

Seemingly reinforcing the above observation that users are seeking (and perhaps not always finding) instructional content development and deployment capabilities, 7 of these top 10 user-supplied criteria might reasonably be understood as pertaining to such functionality.

Table 4 – Top 10 Custom Criteria used in Decision Tool

Feature Name

  1. Ease of Use
  1. Workflow Management
  1. Course Versioning
  1. Editing
  1. Printable Content
  1. Import Third Party Content
  1. SCORM/AICC compliance
  1. Automated Profile Loading
  1. Automated QA

10.      Ability to Develop Database-Driven Interactive Web Pages



Problems with this Study/Possible Areas for Improvements

Top of the list phenomenon

The list of features on which users could compare products and on which they could make decisions never varied in order, and within each of the conceptual categories the features were presented in alphabetical order. Thus users may have succumbed to the general tendency to select from the top of the list. It is possible that this contributed, for instance, in the prevalence of ‘Discussion Forums’ in both comparisons and decisions, though it should also be noted that in this case it is also the most widely supported feature.

Inclusion in decisions is not a full indication of ‘importance’

While the usage of a feature to make a decision does lend it some significance versus features that were not used at all, the Edutools site itself acknowledges in the design of its decision tool that not all features are to be considered of equal importance, and thus allows users to assign weightings in their decisions. Ideally these weightings would also be considered and factored against the feature support across products, but for a variety of reasons, including those of privacy, we have chosen not to store this data.

Different types of users focus on distinctly different sets of CMS features

The analysis presented in this paper would have benefited much from knowing what type of user (Instructor, Institutional Decision Maker, Technical Administrator, Educational Researcher, Student) was using both the comparison and decision tools, as we expect these different sorts of users would focus on quite distinct sets of features. The decision making tool is built specifically to accommodate this diversity of users in its accommodation of weightings. Again, for a variety of reasons, both those of privacy and ones of usability and user-adoption, a decision was made early on in the project to not collect this information.

Appendix 1 – Chart – Number of Products Supporting Features

Appendix 2 – Chart – Feature Support vs. Usage in Decisions