 |
Evaluating virtual outreach services
- The VIVOS project experience.
Alison Yeoman, MSc (Econ), BA, Jan
Cooper, MSc, MSc (Econ), BSc, Dip Soc Sci, Christine Urquhart, PhD, MSc,
BSc, PGCE and Alyson Tyler, MSc (Econ), MA
|
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Click here to see the ppt presentation |
|
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
 |
Abstract
The aim of the VIVOS project was to develop and evaluate methodologies, i.e. sets of methods, for determining the value and impact of 'virtual outreach' information services, in the health sector. Five different projects were recruited initially, with another two added later. Over 130 interviews were conducted, with additional postal questionnaire surveys. Factors impacting on the successful roll-out and continued development of the projects are identified. Conclusions suggest that there is strong support amongst healthcare practitioners for services that can meet their information needs in a way flexible enough to fit in with their daily working practices and patterns and that ongoing information service support is highly desirable.
Introduction VIVOS was a one-year project running from February 2000 to the end of January 2001. It was conducted within the Department of Information and Library Studies, The University of Wales Aberystwyth and received funding from Re:source - the Council for Museums Archives and Libraries. The research team was based at Aberystwyth and collaborated with information professionals from Leicester, Cornwall, West Suffolk, Salford and Trafford, South Humber region, Devon, and North Thames to assess the Value and Impact of Virtual Outreach Services developed at these sites. The VIVOS definition of 'Virtual Outreach Services' was fairly broad and covered any services that enable healthcare professionals to access information without physically coming into the library premises. In most cases this meant providing access to electronic information sources but at one of the VIVOS research sites it involved the dissemination of a hardcopy digest of topical Evidence-Based issues. Development of research skills by the information professionals was seen as an integral part of the project. The project used a variety of research methods, but primarily a qualitative approach was taken to determine the benefits to health professionals of using each service and the way use of such services might be encouraged. Aims and objectives The aim of the VIVOS project was to develop and evaluate methodologies, or sets of methods, for determining the value and impact of 'virtual outreach' information services in the health sector. The findings were intended to inform guidelines for project-management of these services, most of which are in the early stages of development at present. The objectives were: Site selection Five existing outreach services were involved in the project from the outset (in Cornwall, Salford and Trafford, South Humber region, West Suffolk and Leicester). Later, data were collected or provided from an additional two sites (North Thames and Exeter). The sites were selected because they provided a variety of project types and environments, and the evaluation methods used were tailored partly to the needs of the VIVOS project, and partly to reflect requirements agreed with the library service managers at the sites. A specific aspect of each outreach project was identified for evaluation by VIVOS as shown in Table 1.
Table 1. Projects evaluated by VIVOS
Methodology The methods used for the VIVOS study were based on those developed for earlier research projects (Davies et al., 1997; Urquhart and Davies, 1997; Urquhart et al., 1999). The emphasis in VIVOS was on the collection of qualitative data through semi-structured interviews (over ninety percent of which were face-to-face with the remainder conducted by telephone) - supplemented by vignettes (Chell, 1998) and the use of the critical incident technique (Abad-Garcia et al., 1999; Chell, 1998) - and questionnaires. Adapting the methods In keeping with the VIVOS objective of assessing the usefulness of a combination of evaluation methods and because there were specific research questions to be addressed at individual sites, a different 'mix' of methods was used in each case. Details of these methods can be found in Table 2. The North Thames evaluation centred on interpretation of existing data and is therefore not included.
Table 2. Research methods used by the VIVOS team
The challenge was to design survey instruments appropriate to the projects operating at each site whilst maintaining a consistency of approach that would enable analysis of the data across the sites and permit the identification of common themes. Sampling The guiding principle of the sampling process was to use a stratified, randomised sample to avoid bias yet provide a wide range of job-types and work environments from which to build a comprehensive picture of health professionals' use of and attitudes to the virtual outreach services. Table 3 gives a breakdown of the interviews conducted by job role of the interviewee.
Table 3. Number of interviews
conducted with interviewees from different job roles
Response rates Effort was concentrated on obtaining
as large a number of interviews as possible. Given the timescale of the
project (1 year), this meant that little time could be devoted to chasing
up non-responders to the questionnaire surveys. The questionnaire response
rates were generally over 33 percent, but none exceeded 50 percent. The
lowest response obtained was 4 percent (an email survey of medical students
at The University of Leicester). The comparatively low response rates might
be attributed to the length of the questionnaires, but this has to be balanced
by the need to collect data to complement that obtained through the interviews.
Results Essentially, the VIVOS project was a 'snapshot in time', cross-sectional study with the semi-structured, informal interviewing technique enabling a picture to be built up of the factors impacting on the daily working life of healthcare professionals in both hospital-based and community settings. Through analysis of the interviews, researchers identified a range of concerns common to healthcare professionals from across the sites: increasing caseloads, thinly-spread staffing which must service the increasing expectations of the public, lifelong learning, Evidence-Based Medicine, the impacts of clinical governance and audit, broadening managerial roles, new technologies, restructuring of Trusts, and greater emphasis on care in the community. Not only can healthcare professionals feel overwhelmed by their caseloads and commitments, they also have to assimilate a huge amount of information and the potential for information-overload increases all the time with the continuing expansion of IT and the Web. Picking up on the theme of this session, 'the librarian as a gatekeeper', we see health professionals as being confronted with a whole array of open doors and with limited time for exploration. Rather than librarians simply opening additional doors, we see the challenge to be one of careful, rational and considered guidance of health professionals towards doors leading to information that is appropriate, timely and above all digestible. The projects included in the VIVOS evaluation were launched by teams who had identified this challenge and were working towards meeting it in their individual contexts: by offering access to PCs and skills training to groups whose opportunities had previously been limited (for example amongst community-based staff); by expanding access to electronic resources outside working hours and outside the workplace; or by providing tailored information sources with a local focus. Using the right keys to the right doors at the right time As stated earlier, one of the aims of the VIVOS project was to involve the site librarians in the research process and to nurture their research skills. As part of this process, a post-project workshop was held to invite feedback from the site librarians and to investigate project-management issues related to the development of these projects. From the discussions at this workshop and from the results of the individual site evaluations, researchers identified several factors that impacted on the successful practical roll-out of the projects: The 'vision' or goal One of the critical success factors identified at the workshop was the need for the project team to have a clear goal. Although flexibility of approach is necessary to cope with the unpredictable nature of projects that require participation from a variety of stakeholders, there must be an identifiable and attainable long-term 'superordinate' goal (Pinto et al., 1993) or 'vision'. VIVOS set out to establish the extent to which these goals were being attained, to assess levels of user-satisfaction, and to investigate whether the skills and information acquired are being actively channelled into healthcare practice. Motivation The healthcare professionals surveyed displayed a high degree of motivation in the use of electronic resources and for the development of information skills. Encouragingly, this motivation was noted across the spectrum of job-types and working environments and was seen in enthusiasm to attend training courses and in the fact that several interviewees were keen to develop their skills outside working hours. The challenge here is to harness this initial enthusiasm and keep it stimulated whilst at the same time responding to the needs of those participants who are filled with apprehension at the prospect of using a computer. Maintaining the momentum of enthusiasm can be difficult when librarians are faced with complications beyond their control such as the delayed roll-out of IT equipment at practices. Sensitivity to local issues It is important to understand local factors that will impact on the take-up of services or training programmes. Projects at the VIVOS sites had to be flexible, responsive to the existing working environments and sensitive to local need. In Cornwall, training was delivered at individual primary-care practices to overcome the difficulties of travel in a rural county and the content of each session was pitched to participants' level of experience. In South Humber, the editorial board of !evidence matters! use evidence-based resources, the monitoring of media coverage, and feedback from the librarian at the Health Authority to identify 'hot topics' with local relevance to appear in this regular bulletin. Wide-ranging support Virtual outreach projects cannot exist in isolation and benefit from the support of an institutional culture that encourages staff development and recognises that the library has a role in the 'lifelong learning agenda' - i.e. that the library contributes to 'education' rather than simply supplying training courses. In the early stages of setting up a project it is important to have support from all stakeholders including from other departments whose continuing collaboration will be necessary, for example the IT department. Responsibilities must be clear, with service level agreements set up if appropriate. Continuing managerial support is invaluable as the project matures, for example when several members of the same primary-care practice will be absent to attend a three-day training programme, or to ensure the regular supply of updated information to a directory resource like the Pink Book. Library staff will need additional support, including those staff members not directly involved in the project itself. Their awareness and training needs should be identified and met to prevent enthusiasm waning. Meeting the needs of users Healthcare professionals have different levels of experience in IT, and have different expectations and needs. It is therefore difficult, if not impossible, to provide information sources and/or training that will consistently meet the needs of all users. Evidence was found across the projects to illustrate this and for those sites where the evaluation focussed on training programmes it was a particular problem. Although every effort was made in Cornwall to assess the skills-levels of participants before the sessions, attendance was unpredictable and the trainer had to adapt the course content to suit the majority of participants on the day. At Salford and Trafford the decision was taken by the e-STABLISH team to organise groups by primary-care practice and to represent a variety of roles from each practice. The groups attending the course were therefore not streamed according to previous IT and research experience or interests. This approach does have advantages, and several interviewees specifically mentioned that they enjoyed the chance to learn from the experience of their colleagues, however others found that the range of skills-levels was too wide and that the pace of training left them either bored or unable to keep up. Awareness and early identification of benefits Although there was widespread appreciation of the skills and support provided by librarians, it was felt that their services could be more widely advertised. Take-up of new services and training opportunities could be improved by a more proactive approach to alerting people to the potential benefits. Gaining confidence with new technologies does require considerable personal effort and investment in terms of time and motivation. The tangible benefits of making this investment need to be reinforced from an early stage. To encourage engagement, librarians need to capitalise on existing motivation and use a wide range of communication methods to promote services. Interestingly, one of the most effective methods of promoting services appeared to be the recommendation of satisfied users who went back to units and enthused about library services to colleagues. Keeping the doors open Putting projects into practice requires significant effort and collaboration with other professionals and departments. However, a further, and perhaps greater, challenge lies in the maintenance and development of services once their initial impetus has passed. The financial assistance given to kick-start a project may be readily available in the early stages, but the burden of keeping it going or refining it to meet changing needs may fall on over-stretched library resources. Feedback at the post-project workshop contributed to the identification of steps that can be taken to promote the continued success of projects. Process of review and feedback Ideally, a structured process including the stages of testing, evaluation, taking stock and moving on should be followed. This is not always achievable, but identification of factors that signal success and the appraisal of services against these factors, along with monitoring of user needs and readiness to respond to the changing environment all help to maintain the momentum of projects. The VIVOS evaluation contributed to the review process of the participating projects, and the site librarians commented that added value came from involvement in a wider study that raised the profile, and hence awareness, of their projects at local and regional levels. Continuing support for users During the VIVOS interviews, participants revealed that their information-seeking behaviours are fuelled by a combination of: educational requirements, patient care, research, the preparation of guidelines and protocols, and audit procedures. Although motivation for the use of electronic resources was found to be high, simply providing services and training is not enough to stimulate independent information-seeking. The majority of interviewees confirmed that they were most likely to look for information when it was required to satisfy a specific need, for example to complete a coursework assignment. In many cases, lack of independent exploration of resources was related to limited time for searching, to logistical issues such as poor availability of fully-functioning PCs, or to weak confidence in IT skills. This can lead to very spasmodic use of electronic resources, which in turn compounds the problem as users forget the skills they have acquired and lose yet more confidence. Post-training 'top-up sessions' provided in Cornwall proved popular, were well-attended and had the advantage of permitting trainees to return with specific questions once they had had opportunity to practise their skills in their daily work environment. An added advantage of the sessions was that new trainees turned up having heard about them by word-of-mouth. As they attempt to cope on a daily basis with pressures of time and the demands of patient care, users expressed a preference for the convenience of support as and when they need it, for example via a dedicated help-desk. This can place an unpredictable burden on library staff and resources, especially with increasing numbers of users accessing databases from home and outside conventional working hours. Librarians can help to keep the doors to information open around the clock by providing regular training sessions to ensure that health professionals have the skills necessary for successful independent information seeking. Cascading of skills Cascading can be used to widen the impact of a project and give it life beyond the dedicated training sessions. At the Salford and Trafford site participants were expected to cascade skills learnt in training to other staff from their units. In practice cascading was not as extensive as had been intended by the e-STABLISH team although some was successfully achieved. The reasons given at Salford and Trafford and also in Cornwall for not cascading included: low levels of confidence, lack of time, and logistical problems relating to a lack of IT equipment. The lessons learned from this indicate that if trainers do intend a cascading process to take place they should monitor it and perhaps actively pump prime it in the initial stage. Continued support for those cascading the skills would also be a benefit. Continued evolution of the project Virtual outreach projects are not
static and need to respond to the changing environment in which they operate.
This can require significant investment of time and effort as library staff
build on the results of their review and evaluation processes to shape
projects to meet the evolving needs of users. Training sessions must keep
pace as new sources of information become available and as participants'
skills levels and interests develop. Topical bulletins such as !evidence
matters! must retain their focus and individuality to ensure that they
are easily identifiable in the midst of all the other information that
bombards practitioners and are not simply 'added to the pile'. The transfer
of the Pink Book to a Web-based format should make the maintenance process
easier but brings with it a new set of problems in terms of who should
have access to make changes and how its content should marry with that
available through the Trust Intranet.
Conclusions The healthcare practitioners surveyed by the VIVOS project were clearly working under considerable pressure and were at risk of being inundated with information. Outreach services such as those surveyed by VIVOS have an important contribution to make in equipping practitioners with the skills to critically appraise and exploit the resources available, whilst meeting the challenge of guiding users through 'doors' leading to timely, relevant and topical information with a sound Evidence base. However, the initial start-up and continued provision of these services can require significant investment of time and resources to ensure: As the benefits of outreach services become more apparent and necessary, it is hoped that the shared experience of the sites in the VIVOS study will be of use to other libraries seeking to set up similar services. There was strong support amongst the participating healthcare practitioners for services that can meet their information needs in a way that is flexible enough to fit in with their daily working practices and patterns. Many were reaping the benefits of increased skills and access and had found involvement in the outreach projects to be an empowering and rewarding experience, as is illustrated by this comment from a nurse in Leicester: 'And for me it was such a great
achievement, to roll all that together just seemed like something that
I would never ever, ever be able to do......Because I am a doer, I am not
a writer or a thinker, I am a hands on person really, so it was a great
achievement really.'
References - Abad-Garcia, M., Gonzalez-Teruel,
A. and Sanjuan-Nebot, (1999) Information needs of physicians at the University
Clinic Hospital in Valencia, Spain. In L. T.D. Wilson and D.K. Allen, (eds).
Exploring the Contexts of Information Behaviour. Proceedings of the Second
Internation Conference on Research in Information Needs Seeking and Use
in Different Contexts. Sheffield, 13-15 August 1998. London: Taylor Graham.
pp. 209-225.
|
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
 |
|