Section 5
Lessons learnt
Through our experience of the development of programmes for the NCSL, specifically the NPQH and a pilot of the Certificate of School Business Management programme for bursars, we have learnt a number of lessons and have key recommendations for others implementing such programmes.
ULTRALAB's understanding is based on a number of premises outlined in section 1; namely that online community discourse adds value to most programmes. This is based on the belief that discourse within a community context allows participants to collaboratively reflect, develop and share best practice, unlocking the tacit knowledge resident in professional learners.
ULTRALAB have been running assessed online programmes since 1994 when the first online Integrated Course module for a university MA programme was designed and run within ULTRALAB. We have been running online communities since 1993. This report draws on these experiences.
To date (May 2002) over 6,000 online learners and 517 online teachers have been involved in the NPQH programme and 100 learners and 10 online teachers in the Bursars' Development Programme pilot.
5.1 Guidelines for setting up an online programme
At programme inception there is a need to clarify understanding of all the elements of the programme with all members of the programme team, including the project leader and material writers. It is essential to ensure that those designing and developing the online environment are involved in all the early discussions of any of the programme's elements. ULTRALAB's experience with the development of NPQH emphasised the importance of this aspect.
Simultaneously there was rapid development of the NPQH programme, the materials, the content of the face-to-face element and the final assessment methods. By the time of the launch of the new programme, in January 2001, there was little integration of the individual elements of the programme. For example, although online community dialogue was a substantive part of the programme and embedded into the philosophy of the new programme, there was no assessment of the online contributions. It was apparent to learners that there was no requirement to contribute online, as demonstrated by the comment by one NPQH tutor below,
"Horrified after I picked up a discussion 'thread' in Mar01 group that talks of " assessors NOT wanting to see think.com contributions/discussions as evidence" ( Will someone please tell me this is not true before I sceam ?) NB Hard copy print-outs of think.com evidence is as valid, reliable, attributable, as any other material."
(Comment in online community from NPQH online teacher)
Participation in the online community should be part of the assessment process and be regarded as essential; philosophically this gives the message that contributions are of value and online contribution is worthwhile. When the Bursars' Development Programme pilot was developed in January 2002, assessment of the online dialogue was written into the programme at the development stage. Learners understand that this is essential and consequently worked to overcome barriers to communication.
"thank you! Please forgive my pedantry, but I wasn't sure if I was merely confused, or hadn't learnt properly... (aha! another entry for my Journal -or is that cheating?). duz ythis program have spellcheck?"
"Don't really want to post this, seem to be hogging the show, but if you don't ask you don't learn.., HEY, I've found the spell check, in Preview! (Another L-J entry, heh heh heh...)"
(Two comments from the same learner in Bursars Count noticeboard)
Demonstration of the value held by community can be made in a number of ways, including the commitment to participate by those running the programme.
The elements of the NPQH programme were seen as standing in isolation. For example there was no extension of discussions held on face-to-face days into the online community dialogue. Thus an opportunity was lost to provide an integrated approach for the learners. The bursar pilot has structured the programme of hotseat guests so that a presentation is made at a face-to-face event; the slides from the presentation are made available to learners who are then able to ask questions in the community hotseat. This gives learners more time to think about their questions and the speaker can answer when convenient. This integration of the face-to-face meetings with the online community is a welcome progression, improving the relationship of the online elements with the face-to-face.
It is essential to ensure that the programme materials can integrate with online dialogue. The NPQH writing team had little experience of online learning when the questions to be discussed were written, hence the activities specified in the modules for online community dialogue were not always suitable or appropriate. Teachers reworded them or replaced them with other activities designed to stimulate discussion (see section 3).
For the development of successful online community discussion it is essential that the underlying programme philosophy is in symmetry with participation and dialogue. If the programme exists in isolation, as in the case of LPSH, then the online element is likely to fail. Within the NPQH programme, the materials' website was designed to provide flexibility, thus the underlying database and system of hyperlinks enabled the online learner to 'follow a journey' through the materials. Online learners are able to use the materials website to access both the programme materials and resources and up-to-date information from external web sites, for example when studying the issues around setting a school budget, the website contained background information, links to organizations, such as the Audit Commission, and spreadsheet models that can be downloaded and tailored to individuals needs.
Each NPQH module emphasised reflection and 'cued' online learners for the reflective experience. The materials website consistently linked back to the online community. This gave online learners the opportunity to develop a pedagogy based on dialogue, discussion, collaboration and sharing best practice within the communities to which they belonged. Thus the materials website was consistent with the changed philosophy of the NPQH programme; this was not a managed learning environment in which control resided in the programme team but an information and learning tool through which the pedagogical thinking residing in the programme (self-directed learning, continual professional development, collaboration, sharing best practice and criticality) could be best implemented.
There is a need for any programme to consider the potential learners to ensure that their learning will be enhanced by access to a learning community. If the philosophy of the programme is reflective and self directed our evidence is that learning will be enhanced (see section 3).
5.2 The structure of the online learning environment
The online space for any programme should consist of an overarching space and spaces for learning, which will have distinct features, to match the style, aims, objectives and numbers involved in the programme. Both these types of spaces are discussed in this section.
5.3 Overarching Area
All programmes, including NPQH and the bursars' Certificate of School Business Management pilot have demonstrated the need for an overarching area where all online learners meet. This was established in the early planning of the online aspects of NPQH. This 'central' place (Virtual Heads and Bursars Count) provided crucial administrative functions. For example all the necessary programme documents were available for online learners to download in Virtual Heads, both communities provide a 'noticeboard' for online learners and teachers to highlight issues, problems and notices.
Image 3.1 (in section 3) shows the overarching area for NPQH and Image 5.1 that for the pilot Bursars' Development Programme.

Image 5.1 The overarching area for the pilot Bursars' Development Programme
All those involved in a programme belong to the overarching community. This provides for essential access to a feedback space for the online learners.
5.4 What should an overarching area contain?
Whilst programmes will vary and have distinct 'flavours' there are some generic features which can be identified and should be in each overarching area.
Experience shows that these overarching spaces are well used by online learners who give valuable feedback on all aspects of the programme, including design, implementation and materials. The examples below give two of the many types of feedback from Virtual Heads.
"I'm a new member to NPQH and was relieved to read that other people are finding it difficult to get online - I thought it was me/my eldery computer. Finally made it tonight although I had o make a cup of tea whilst waiting for photo to install....No doubt it will become clearer in time. I look forward to contributing to discussions, ask Hotseat questions etc but so far find it a little intimidating. "
"What has changed? Tonight for the first time everything worked without me having to try twice at all and i was on for nearly half an hour. Well done! Is it the time or a fluke. I hope not because it certainly made a difference to what i could do in the time. I will let you know if it continues."
(Comments from online learners in Virtual Heads feedback)
A consistent approach should be adopted in these spaces, few changes being made to ensure that online learners can find the information they need with as little waste of time as possible. Within Virtual Heads, from the programme inception in January 2001 the area contained a noticeboard, where both online learners and teachers could raise issues, links to documents and external sites (including the NPQH materials web site) and the hotseats.
5.5 Programme learning communities
Early learning from Talking Heads emphasised the importance of limiting the number of places those contributing online needed to visit to contribute. When the number of places where contributions can occur is reduced, then less members are viable. Talking Heads participants polled for feedback frequently mentioned too many places and too many discussions, as demonstrated by the comment below from a member of Talking Heads,
"I cannot find the discussion I started to find out if anyone has commented ..... there are simply too many places to go to and too many overlapping discussions."
(Comment in personal communication from Talking Head participant to a facilitator)
In the development of NPQH, the project leaders wished to replicate the face-to-face experience online and thus created a large number of private communities. For example in January 2001 online learners on the NPQH programme were expected to discuss their learning in a space designed for one online learner and one online teacher. This mirrored tutorials in a traditional learning situation These areas were called Summaries of Learning and the anticipation was that online teachers would interact with online learners on a regular basis to add advice, expertise, support and encouragement to the online learner. Each NPQH online teacher had 12 to 15 online learners and therefore 12 to 15 Summary of Learning areas. This led to unfortunate replication of the problems identified by Talking Heads participants.
Some participants produced active and impressive Summaries of Learning. Despite the success of this minority, a large number of online teachers were still embracing their capabilities with the tools and did not cover all the administrative tasks required by the software. For example, they omitted to invite participants to the spaces or created the areas in the wrong places. When this area was created for each learner this led to confusion, due to the number of online spaces created (over 2,500) and the impossibility of ULTRALAB facilitators or regional co-ordinators checking to correct errors. A more serious problem, however, was that in cases where participants provided impressive Summaries of Learning these were locked in two-member community isolation, unshared and uncelebrated. This was completely contrary to the philosophy of both community and the programme, which emphasised collaboration and shared learning.
Following advice from the ULTRALAB facilitation team it was agreed to experiment with combining the Summaries of Learning of each online teacher, creating Shared Learning Spaces of between 12 and 15 online learners with one online teacher. The Shared Learning Space provided localised communities (online tutorial groups) in which NPQH online teachers encouraged, stimulated and orchestrate meaningful debate between online learners and gave shape to the learning. This smaller shared learning group is supported by some face-to-face tutorials and learners are encouraged to meet and visit each others schools. Within the online space it was anticipated that online learners would 'collect' comments, discussion points, questions and answers from other parts of the online environment that were relevant and key to their own learning and share these with their group. By transforming these Summary of Learning areas into Shared Learning Spaces, allowing other members of the group to comment, the programme ensured that there was a real purpose for these areas and that the learning which took place could be shared. These spaces have become the online learners' principle record of formative learning.
By May 2002 Shared Learning Areas had become lively and vibrant, especially in the Access stage of the programme. Within these areas the personality and tutoring style of online teachers, who have embraced the environment, shines.

Image 5.2 A tutor's customized summary of learning group space
Participants have been keen to extend the remit of these areas initiating all sorts of discussions including those surrounding the modules but at the expense of participation in the Module Discussion Groups.
Virtual Heads Module Discussion Areas were designed so that discussion could take place with a large number of online learners. Discussions in these areas surrounded the programme modules, which were regionally based and managed and were cross phased. These communities aimed to provide online teacher-led, predetermined discussions that were heavily focused on the specified activities within the units. It is in this forum that online learners were expected to share their pragmatic, school based experiences and online teachers were expected to enhance those experiences by drawing the online learners into a more reflective mode. Despite an increase in the membership of these communities, from 54 in January 2001 to 300 in January 2002, there remains in May 2002 only patchy evidence of discussion in these areas and a lack of clarity of purpose as the Shared Learning Spaces gradually extend the discussions from the original brief into all aspects of the programme.
5.6 Key reasons for lack of participation in module discussion areas
A number of reasons can be identified for the module discussion areas attracting very few participants:
Despite lack of participation in these areas by November 2001 a number of online teachers were experimenting with both the questions and the online environment by offering less structured discussions, combining a number of questions and adding social dimension to the communities. This stage is important as it marks the phase when the online teachers became so confident with the technology that they used their expertise as deliverers of learning to experiment in the environment. However a real purpose for the Module Discussion Areas is still sought.
The Bursars have two communities, the Bursar's Count community, to which all learners and tutors belong and their Tutor Group which consists of about 10 learners and a tutor. The tutor groups only have a noticeboard and a learning journal for each candidate. It is in this forum that online learners are expected to share their pragmatic, school based experiences and online teachers were expected to enhance those experiences by drawing the online learners into a more reflective mode. The tutors could use their time more effectively because they could see at a glance who had recently contributed to their learning journal and therefore could respond accordingly. Learners could see what the other members of their group were learning and they were able to comment. They could also see how their work was progressing compared with others in their group.
This smaller group is supported by some face-to-face online tutorials and online learners are encouraged to meet and visit each others schools. The online discussions here are focused on the learning that each online learner has recorded. It was anticipated that online learners will 'collect' comments, discussion points, questions and answers from other parts of the online environment that are relevant and key to their own learning and share these with their group. These form an online learner's journal, which is shared and discussed online with their tutor and which is the principle record of the formative learning.
The pilot for the Bursars' Development programme has yielded vibrant online spaces from the outset. There are several reasons that may contribute to this success:
The important lesson from the above account of the development of the structure of NPQH and Bursars is that a clear purpose for each community is essential. The purpose needs to be expressed so that online learners are clear as to the expectations of them in each area created. In Cohort 1 of the NPQH learners were encouraged to set up Learning Circles, but given the number of areas online learners were able to engage in (DfES, NCSL, Virtual Heads, Summary of Learning, Learning Journals, Learning Circles) this area lacked a strong enough purpose to engage learners. Similarly the Module Discussion Areas are still in May 2002 seeking a clear purpose which is not covered by other online spaces and which is strong enough to engender discussion.
5.8 Integration of a programme
Most online programmes have some element of stand alone programme materials. For the programme to demonstrate value placed on participation and collaboration it is essential that the materials do not stand in isolation. Value was demonstrated within the NPQH programme by ensuring that access to the programme materials was via a hyperlink from within the communities. This allowed online learners not only to use one ID and password (that used to access the community software) but importantly, emphasised the importance placed on the community and discussion elements of the programme, because the online programme materials could only be accessed by passing through the community space.
In addition the materials website in NPQH enhanced the philosophy of participation and collaboration by continually linking back to the communities and encouraging learners to comment. Image 5.3 below taken from the NPQH online materials site demonstrates this link.

Image 5.2 Signposting from the materials website
The structure of the materials site emphasised the reflective aspect of the programme as demonstrated by image 5.4 below, taken from the NPQH Access materials site shows.

Image 5.3 Linking to the summary of learning from the materials website
This continually reinforced the programme philosophy. In terms of the diagram in section 1 (Table 1.1) this meant that the NPQH programme firmly placed itself towards the constructivist side of the continuum. In contrast the Bursars programme materials site was developed as a self contained, divorced, structured task environment, sitting firmly on the managed learning side of the continuum. The result of this was that online learners, who were using the community for one month before the materials site was developed were familiar and comfortable with participating and sharing:
"Leaping ahead a bit, I know that shall miss this community when the course is over and I know from conversations at Dunchurch, that others will, too. I wonder if it might be possible, for those who wish to, to stay on as members and perhaps contribute and help with subsequent cohorts? It seems a good way of putting something back whilst also consolidating our own learning. Do you think that might be likely? :) "
"Not related to the course at all really, but can anyone tell me what policies they have in place re: risk assessment. This is something that we as a school need to look at and I would very much appreciate any pointers. Thanks."
(Comments in Bursars Count noticeboard)
These learners therefore found the materials site disjointed and difficult. ULTRALAB emphasis that this was not only a structural issue but more fundamentally a philosophical one.
As the move is made towards self reflective, self paced study the pace of the course needs to be considered. Within the NPQH programme online learners were expected to cover aspects of the programme in whichever order and at whatever time they wished. As with any truly flexible programme NPQH participants were able to make use of prior knowledge and ignore those elements where their learning was sufficient if they wished. This resulted in a programme where the online learners could be studying any aspect of the programme at any time. Pacing a programme this flexible was a challenge. In January 2001 the NPQH programme team had a vision, taken largely from a face-to-face environment, of a two weekly cycle of activities that participants could join in when and if they wished. This did not translate well to the asynchronous online environment, where one of the main advantages is the ability to slip time. Discussions closed before all online learners had a chance to participate and those who chose to do the programme in a nonlinear fashion found the areas they were interested in already discussed, summarised and closed. In September 2001 the discussions surrounding the materials were placed online at the start of the programme allowing online learners to add discussion points when they liked. This however meant the discussions were slow and disjointed, dialogue rarely developing and was one of the key elements in the failure of the Module Discussion Areas (see sections 3 and 4) In the more flexible Shared Learning Spaces aspects of the materials can be integrated into the discussion when they affect the online learners to the rhythms and demands of the school year can be integrated into the course, for example the programme material on managing a budget can be discussed at the time when the online learners are concerned with budgets in their school, with the online teacher pointing to the archive of, or current 'live' expert discussion (hotseats) on budgets. This enables a true integration of all elements of the programme.
5.9 Lessons learnt following 12 months of iterative development of NPQH
The learners following the Access programme use the online areas more heavily. This is because the emphasis in the Development programme is on school based work, which means that participants have less need for online collaboration, whereas in Access the emphasis is on sharing best practice and learning. This emphasizes:
5.10 Key recommendations
5.11 Facilitation of a programme
ULTRALAB have established in their early work (www.ultralab.net/papers) that facilitation is essential for any online community to thrive. Facilitation is a skilled activity and those facilitating any online programme need to be aware of the aims, objectives and philosophy of the online programme.
In the design of any programme knowing who the target online learners are is essential. In January 2001 28% of those applying for NPQH filled in the online application form, whereas by September 2001 over 60% applied online. From this information the ULTRALAB facilitators were able to anticipate that the level of access to the internet and the higher level of ICT skills amongst the later intake would make the online community aspects more successful. This, combined with the iterative process which changed the programme (see section 4.4) resulted in the Second cohort having a much more positive experience online. With the Bursars pilot programme online learners were already using ICT in their work, (although many had no experience of the internet) which meant, as anticipated, the technical barriers were few. In addition for those participating in the Bursars programme there was a degree of isolation from other colleagues being only one Bursar per school. The community was effective in removing this isolation.
"Great idea to provide current Bursar's Count members with some form of on-going support / links. Everyone I'm sure finds the days together both socially enjoyable and professionally rewarding. We all need to adopt a degree of continued professional development after the Certificate and I think a Bursars Count "Old Boys" group could be a very popular and supportive arena. Bursars Count has such a specific momentum developing it would be a shame not to harness the NCSL's success."
(Comment in Bursars Count noticeboard)
5.12 Reasons for success: dynamics of a group (bursars)
Hence one of the key elements in success of an online programme is the dynamic of the group the programme is designed for. In the case of bursars to following factors combined to ensure that Bursars Count was successful and the community valued:
ULTRALAB's experience is that at least two facilitators per programme are essential. Not only does this allow facilitators to support each other and allows for exchange of good ideas, but on a practical level ensures the community is covered over holiday periods and that the communities are not left untended for more than 72 hours. Two facilitators becomes especially useful when training of online teachers is required.
It is essential that facilitators are kept fully informed of programme developments, organisation and content as well as the roles of others involved in the project. Within the development of NPQH, ULTRALAB facilitators were not involved in, or given any information about assessment process as this was seen as not in their domain. If, however, they had been involved at an early stage they would have strongly advised that some aspect of online community be involved in the assessment. Only after three iterations of the programme is this now being 'written' into the assessment process. In the example of the Bursar's programme those responsible for running the programme saw the developments in NPQH and were therefore willing to take advise from ULTRALAB facilitators thus building on experience gained.
5.13 Online teachers
ULTRALAB facilitators are experts online, their expertise is in online facilitation and not in the subject of the programmes. As such any programme requires online teachers. Online teachers; make the online space welcoming and relevant to the programme participants needs, ensure learning takes place through provocation/questioning/posing of specific questions, direct the learning and the sharing of key learning points, establish personal contact with programme participants if they encounter difficulties or if they do not appear to be present a short period of time after induction (for example 10 days) and allow online learners to display their expertise online at the start - working online is hard enough without having to engage initially in heavy academic discussion. ULTRALAB recommend that starter discussion seek to draw on the online learners area of expertise, making it easy to contribute but ensuring that all contributions are relevant.
One of the key roles of the online teacher is the setting up of the online community space. Although the design of the space will have been agreed before the programme starts ideally the communities should be established by the online teachers so that they feel 'ownership' of the space. Online teachers should personalise the space, For example they should have their pictures or icons published and write about themselves if the software allows. This helps reduce the impersonality of the online environment and a few sentences of introduction can help reduce any reticence to communicate. It also encourages learners to do the same. This is the online equivalent to wearing a name badge at a conference or programme and encourages the building of relationships. There are a number of givens in establishing the online community space outlined below;
5.14 Key recommendations
With both the NPQH and Bursars programme the online teachers had very limited experience working online. In the case of NPQH 516 online teachers needed to understand the environment in which they were to work in a very short period of time. NPQH online teachers first saw the online environment in which the programme was to take place in October 2000 when their usernames were created. The first cohort of online learners were scheduled to come online and start the new programme in January 2001. This meant an intensive training programme was essential.
Philosophically ULTRALAB believe in empowering people to take control of their own environment and do not believe in a model of dependency. With these aims in mind the aim of any training programme developed for online teachers is to empower them to run the programmes and to allow for the disengagement from ULTRALAB facilitation.
Developing from ULTRALAB experiences with both NPQH and Bursars training ULTRALAB believe that training for online teachers new to working in an online environment should take place not only face-to-face but in the online environment in which the programme is to take place. An online training programme should ideally consist of;
1 day face-to-face training with a full computer suite and a digital projector
4 weeks online training with a commitment to contribute for 2 to 3 hours a week
The online training programme should allow online teachers to experience being taught online, and thus through skilled and active facilitation allow them to understand and model behaviour for naive users. Thus the emphasis of any training programme should not be on the technology but on the philosophy and understanding of teaching and learning online. Commitment to participate is essential. A course modelled on the guidelines above was developed for NPQH centre manages in September 2000, but with little commitment and interest on the part of some managers participation was low. NPQH research has shown that in those regions where the centre managers undertook the course and actively participated online teachers have had less problems and thus the experience of learners has been generally rewarding.
Online teachers need an initial face-to-face induction programme followed by an online extended course. The induction needs to clarify the course philosophy, not simply concentrate on the ICT element. Although initially the overcoming barriers to ICT may appear to be the strongest basis for training programmes, any training based solely on this outcome will not lead to empowerment or arm online teachers to work in the environment. For this reason it is essential that ULTRALAB facilitators are involved in the programme developments so that training can be integrated into the programme requirements and the programme philosophy can be integrated into the training. Any training activities which do not mirror the requirements of the programme and do not lead to understanding and reflection on the use of the community environment will leave those being trained unaware of the potential of the environment. In the early training stages of NPQH not enough attention was paid to the change in the programme structure and philosophy. Online teachers were not therefore left aware of the full implications of a self-study, self-directed, flexible programme. If the ULTRALAB facilitators had been involved in the whole programme, rather than simply the online aspects, many issues could have been addressed at an earlier stage and a number of problems could have been avoided.
After tutors had received their initial NPQH training they were invited to join the Good Practice community which was established in February 2001. This space was where the ULTRALAB facilitators could communicate directly with all NPQH tutors, although support via 'back channels' (email, telephone) is frequently sought. Advice on using the tools and examples of a Module Discussion Area and Summary of Learning Group were provided. Online teachers could ask the ULTRALAB facilitation team questions, initially these were mainly technical but soon addressed other issues such as participation.
"I need to set up some easy way of flagging to all my tutor group that there is something they must look at without sending stickies with detailed instructions to each."
(Sample comment from online teacher in "Good Practice" community, 26th July 2001)
Since October 2001 it has started to become a channel for online teachers to share good ideas and innovations. This is a stage towards community ownership.
"At our recent tutor meeting, our tutors wanted to know how we can evaluate what learning has gone on from online activities. Has anyone any ideas?"
(Sample comment from online teacher in "Good Practice" community 11th November 2001)
The online teachers on the Bursar programme also had a community which was relatively small but used extensively by organisers and online teachers for communication, dissemination of information and sharing of ideas.
As online learning is still new many online learners will not have clear expectations from a programme delivered largely online. This being the case an initial face-to-face element is useful to set out expectations of all parties. In an environment which is self-study and self paced, emphasising the knowledge residing in the participants expectations could be set via a negotiated 'learning contract' between parties, but the nature of such a contract will depend on the philosophy, cost and length of the course.
As with any participative programme there should be a clear code of conduct that generates appropriate expectations for participation and reciprocity. This should be understood and agreed by programme leaders, online teachers and learners.
Materials should integrate all aspects of the programme, although for naive participants an ICT handbook may be necessary. As the use of online programmes spreads this will cease to be necessary.
It is vital that the holistic nature of the course should be stressed to online learners and not simply the ICT element. Although ULTRALAB believes that learning should be delightful it also needs to be meaningful. Thus, it is important to demonstrate that community has added value and provide examples which stress sharing and developing good practice.
Hands on experience is desirable because ULTRALAB experience has demonstrated that a successful first login is vital. Evaluation after the first cohort of NPQH learners recognised the desirability of learners having logged into the learning environment as a part of their induction process which helps to 'debug' any initial problems. With the Bursars programme this was not possible as online learners were given only one hours demonstration of the system, but this was combined with a discussion of the purpose and philosophy of the online learning community which is an essential component of any demonstration.
Key recommendations