Good practices related to the creation and management of collections of interactive exercises.

The creation of interactive exercises is a complex process. It needs

1. technical expertise in the particular system, probably some programming language and web design;
2. pedagogic design of the learning materials, particularly when it comes to generating random items; and
3. a sensitivity as a teacher in the writing of feedback and a knowledge of common mistakes and how to respond to them.

It is likely that teams will be needed to develop high quality collections of interactive exercises, particularly for large cohorts or high stakes situations.

Design/selection of systems

An individual teacher may have little choice over which system is available to them. However, they should consider the following issues.

• Many systems accept an algebraic expression as an answer. For example, a polynomial, equation, set or matrix. While multiple choice questions may still have a place, their widespread use is more a consequence of the ease with which they can be implemented at a technical level.
• How does the user get their answer into a machine? Is a strict syntax required, or are other input tools available?
• How is mathematics displayed to students? Are standards adopted? Does the user have the ability to adapt the output format, e.g. size?
• There is a great variety in the extent to which systems permit adaptive testing, i.e. the selection of one exercise based on the outcomes of the current one. The design of adaptive exercises is surprisingly difficult, hence expensive, and a simpler approach may be more cost effective.
• Students should be given an opportunity to review their work between them typing it in and it being assessed. This is particularly important where a one-dimensional input format (e.g. strict syntax) is later displayed as a traditional two dimensional string.
• Are repeated attempts permitted/encouraged?
• The designer must consider any special needs, allowing the re-sizing of fonts and ensuring any special interaction elements used by students (e.g. applets) are useable by all intended students.
• What tools are available to them to manipulate mathematical expressions, e.g. computer algebra, both in generating questions, establishing properties of the answer, and giving feedback which incorporates calculations based on these properties?

Design of learning materials

• Interactive, web-based exercises, are most useful in assessment and practice of basic mathematical skills. Obviously, interactive web-based exercises are not suitable for all problems in mathematics education.
• Most current systems are limited in the extent to which the can assess steps in working. An exercise usually should involve a single method or topic of mathematics, without requiring too many steps to obtain the answer.
• Under what conditions does the student use these materials? Does the designer assume the student has access to technology, e.g. a computer algebra system, or is such technology forbidden?

Creation

Randomly generated items

It is common practice to randomly generate versions of interactive exercises for each student. There are two ways to create random items for students.

1. selecting items from a list;
2. inserting random parameters within an item.

Both techniques can be used simultaneously.

Random questions allow meaningful practice of versions of the "same" question. Of course, "same" to the teacher may be quite different from the students' point of view. For example, in a factorization question, changing the numbers and variables makes the question look completely different to a student new to the subject.

Students regularly say the worked solutions are the most useful part of the interactive exercises. They often look at these first, before asking for new versions to practice. This is an effective way to learn mathematical skills. Hence, the exercises should provide detailed step by step solutions to the problems which also depend on the algorithmic parameters thus giving a solution to the problem instance at hand. In practice mode and after a test students look at the solutions. These help them to learn, or simply remind them of, the methods involved in solving that problem.

The level of detail required by the particular student is encoded within the worked solution. Hence, if the random parameters cannot be sensibly varied without needing a very different worked solution, then we are probably dealing with a different question. If such variation is desirable, switch to a different question and randomly select from a list.

• Start with the worked solution. This provides the level of detail appropriate for this groups of students.
• Create random parameters which preserve this worked solution.

Authors must be very careful to have the possible value set of the randomized parameters meaningful. Avoid too easy or difficult versions or even versions that have no solution at all.

• Illustrative images and graphs help to understand the question setting better. The graphs can also be generated dynamically from the algorithmic parameters.
• The system must keep track of which version is given to each student.

Also, problems that ask the learner to identify some property of a given picture are useful for developing conceptual understanding of a mathematical topic.

Establishing properties of students' answers

• Some assessment systems, such as STACK, allow sophisticated analysis methods of the student's answer and give

feedback accordingly. For example, a usual mistake can be recognized from the answer and then give specific feedback and hint on hot to correct that mistake.

Deploying and testing

• Items must be tested thoroughly, with correct and incorrect answers across the range of parameters.
• Authoring reasonable and error free exercises requires programming and debugging skills of mathematical programming languages.
• Authors usually have notes or tested snippets of reusable code.

Management

• Managing a large collection of exercises requires the use of suitable meta data. At least author, date of creation, mathematical topic, some type of difficulty or intended school level help later to find suitable exercises.
• Using a standard classification specifically designed for educational purposes, such as a national curriculum, or the so called MathTax taxonomy sets common and documented classification scheme for the material.
• Some assessment systems, such as MapleTA, have rather limited meta data support. They might provide some additional information fields to be entered but it is up to the users to coherently use those. Another option is to encode some information to the question names, which again requires agreed upon rules by the user group.
• Different assessment systems provide different ways of organizing the exercises into collections and sub collections. With careful use of these facilities a reasonable structure can be achieved to the collection. For example trying to avoid too large or too small sub collections. (This also depends on the system's capabilities of restructuring the collection.)
• Having a standard meta data format, e.g. LOM, supporting repository for organizing a large question bank is useful. Most of the meta data can be automatically generated from the content, though depending on the structure and meta data of the original collection. A challenge is to provide a feasible connection between the assessment system and the repository.
• Authors should choose a licence for their work, for example under creative commons.

Analysis of results

To close the feedback loop, analysis of activity should take place.

• Some systems can automatically calculate statistical measures of, e.g., question validity. This should also examine whether particular choices of random parameters are significant. E.g. the Rasch model from item response theory or Cronbach's alpha from classical test theory.
• Some systems can automatically generate a profile for each user.
• Thoughtful analysis of students' attempts can inform the design of better feedback for use in subsequent years.