Evaluating virtual outreach services - The VIVOS project experience.

Alison Yeoman, MSc (Econ), BA, Jan Cooper, MSc, MSc (Econ), BSc, Dip Soc Sci, Christine Urquhart, PhD, MSc, BSc, PGCE  and Alyson Tyler, MSc (Econ), MA
Department of Information and Library Studies - University of Wales
Aberystwyth Llanbadarn Campus - Aberystwyth, SY23 3AS (United Kingdom)
alison.yeoman@cwcom.net, cju@aber.ac.uk

Click here to see the ppt presentation


The aim of the VIVOS project was to develop and evaluate methodologies, i.e. sets of methods, for determining the value and impact of 'virtual outreach' information services, in the health sector. Five different projects were recruited initially, with another two added later. Over 130 interviews were conducted, with additional postal questionnaire surveys. Factors impacting on the successful roll-out and continued development of the projects are identified. Conclusions suggest that there is strong support amongst healthcare practitioners for services that can meet their information needs in a way flexible enough to fit in with their daily working practices and patterns and that ongoing information service support is highly desirable.



VIVOS was a one-year project running from February 2000 to the end of January 2001. It was conducted within the Department of Information and Library Studies, The University of Wales Aberystwyth and received funding from Re:source - the Council for Museums Archives and Libraries. The research team was based at Aberystwyth and collaborated with information professionals from Leicester, Cornwall, West Suffolk, Salford and Trafford, South Humber region, Devon, and North Thames to assess the Value and Impact of Virtual Outreach Services developed at these sites. 

The VIVOS definition of 'Virtual Outreach Services' was fairly broad and covered any services that enable healthcare professionals to access information without physically coming into the library premises. In most cases this meant providing access to electronic information sources but at one of the VIVOS research sites it involved the dissemination of a hardcopy digest of topical Evidence-Based issues.

Development of research skills by the information professionals was seen as an integral part of the project. The project used a variety of research methods, but primarily a qualitative approach was taken to determine the benefits to health professionals of using each service and the way use of such services might be encouraged.

Aims and objectives

The aim of the VIVOS project was to develop and evaluate methodologies, or sets of methods, for determining the value and impact of 'virtual outreach' information services in the health sector. The findings were intended to inform guidelines for project-management of these services, most of which are in the early stages of development at present.

The objectives were: 

  • To extend and refine existing value and impact methodologies for health-information use
  • To assess the usefulness of multiple methods of evaluation (quantitative and qualitative), in both determining progress, and identifying remaining barriers to the development of virtual library and information services
  • To evaluate the effectiveness of various training approaches for virtual outreach health information services for community staff
  • To develop guidelines on the methods, and sets of methods, most suited to particular research questions
  • To develop research and evaluation skills among information practitioners by active involvement in the project.
  • Site selection

    Five existing outreach services were involved in the project from the outset (in Cornwall, Salford and Trafford, South Humber region, West Suffolk and Leicester). Later, data were collected or provided from an additional two sites (North Thames and Exeter). The sites were selected because they provided a variety of project types and environments, and the evaluation methods used were tailored partly to the needs of the VIVOS project, and partly to reflect requirements agreed with the library service managers at the sites.

    A specific aspect of each outreach project was identified for evaluation by VIVOS as shown in Table 1.

    Table 1. Projects evaluated by VIVOS
    Site Subject of evaluation

    A database training programme run by Cornwall Library Services, originally for staff in community hospitals but later encompassing primary-care practices.


    A feature of the Trent Futures project giving remote access to NISS Biomed databases through the issue of Athens passwords to staff at Leicester Royal Infirmary. The VIVOS team also surveyed staff and students from the Faculty of Medicine and Biological Sciences at The University of Leicester who have access to the service through the academic network. (Trent FUTURES Project, 1999)

    Salford and Trafford

    A three-day training programme, for the e-STABLISH project which aimed to provide primary-care and community healthcare staff at twelve selected sites with access to evidence-based sources via PCs installed at their practices. (Farrell et al., 1999)

    South Humber

    The bulletin !evidence matters!, a regular digest sent out to local primary-care and community staff alerting them to topical issues in clinical effectiveness and evidence-based practice. The investigation also looked at access to the CINAHL database and the information-related problems faced by nursing staff. 

    West Suffolk

    The Pink Book developed by library staff at the West Suffolk Hospitals' NHS Trust. This was originally a hardcopy directory of information for primary-care clinicians but is now available as a CD ROM with a Web-based version in preparation. The local focus has remained but the content has grown to reflect changes in practice and now includes other features such as guidelines and protocols. (Hunter and Lockyer, 1999; Hunter et al., 1998)


    The Website set up by Exeter Medical Library.

    North Thames

    Additional data analysis for an existing database access project.


    The methods used for the VIVOS study were based on those developed for earlier research projects (Davies et al., 1997; Urquhart and Davies, 1997; Urquhart et al., 1999). The emphasis in VIVOS was on the collection of qualitative data through semi-structured interviews (over ninety percent of which were face-to-face with the remainder conducted by telephone) - supplemented by vignettes (Chell, 1998) and the use of the critical incident technique (Abad-Garcia et al., 1999; Chell, 1998) - and questionnaires.

    Adapting the methods

    In keeping with the VIVOS objective of assessing the usefulness of a combination of evaluation methods and because there were specific research questions to be addressed at individual sites, a different 'mix' of methods was used in each case. Details of these methods can be found in Table 2. The North Thames evaluation centred on interpretation of existing data and is therefore not included.

    Table 2.  Research methods used by the VIVOS team
    Site Interviews conducted  Questionnaires despatched Use of vignettes Use of critical incident technique
    Cornwall 26 - Yes No
    Leicester 35 175 No Yes
    Salford and Trafford 17 43 Yes No
    South Humber 22 - No Yes
    West Suffolk 37 100 No Yes
    Exeter - 200 No No
    Total interviews/
    137 518    

    The challenge was to design survey instruments appropriate to the projects operating at each site whilst maintaining a consistency of approach that would enable analysis of the data across the sites and permit the identification of common themes.


    The guiding principle of the sampling process was to use a stratified, randomised sample to avoid bias yet provide a wide range of job-types and work environments from which to build a comprehensive picture of health professionals' use of and attitudes to the virtual outreach services. Table 3 gives a breakdown of the interviews conducted by job role of the interviewee.

    Table 3. Number of interviews conducted with interviewees from different job roles
    Job Role of Interviewee Number Interviewed
    PAM (Physiotherapists/Occupational Therapists/Nursery Nurses/Hospital-Based Social Workers/Dietitians/Scientific and Technical Staff/Sonographers/Radiographers 22
    Nurse 21
    Senior Clinical Practitioner (Consultants/Dentists) 13
    Community-Based Management (Practice Managers/PCG or PCT Managers) 12
    Community Nurse (Practice Nurses) 10
    Clinical Practitioner (Speech Therapists/Pharmacists/Psychologists/Radiologists) 9
    Primary Care Clinician (General Practitioners) 9
    District Nurse 8
    Health Visitor 8
    Administration staff (Practice Receptionists/Resource Officers/Clerical Officers/Secretaries/Information Officers/Ward Clerks/Team Clerks) 7
    Training-Grade Clinician (SHOs/SPRs/VTs/PRHOs) 5
    Midwife 4
    Health-Related Research Officer 2
    Hospital-Based Management 2
    Teaching Post 2
    Community-Health Management (Community Health Council representative) 1
    Nursing Management 1
    Research Management 1
    Total Interviews 137

    Response rates

    Effort was concentrated on obtaining as large a number of interviews as possible. Given the timescale of the project (1 year), this meant that little time could be devoted to chasing up non-responders to the questionnaire surveys. The questionnaire response rates were generally over 33 percent, but none exceeded 50 percent. The lowest response obtained was 4 percent (an email survey of medical students at The University of Leicester). The comparatively low response rates might be attributed to the length of the questionnaires, but this has to be balanced by the need to collect data to complement that obtained through the interviews.


    Essentially, the VIVOS project was a 'snapshot in time', cross-sectional study with the semi-structured, informal interviewing technique enabling a picture to be built up of the factors impacting on the daily working life of healthcare professionals in both hospital-based and community settings. Through analysis of the interviews, researchers identified a range of concerns common to healthcare professionals from across the sites: increasing caseloads, thinly-spread staffing which must service the increasing expectations of the public, lifelong learning, Evidence-Based Medicine, the impacts of clinical governance and audit, broadening managerial roles, new technologies, restructuring of Trusts, and greater emphasis on care in the community.

    Not only can healthcare professionals feel overwhelmed by their caseloads and commitments, they also have to assimilate a huge amount of information and the potential for information-overload increases all the time with the continuing expansion of IT and the Web. Picking up on the theme of this session, 'the librarian as a gatekeeper', we see health professionals as being confronted with a whole array of open doors and with limited time for exploration. Rather than librarians simply opening additional doors, we see the challenge to be one of careful, rational and considered guidance of health professionals towards doors leading to information that is appropriate, timely and above all digestible.

    The projects included in the VIVOS evaluation were launched by teams who had identified this challenge and were working towards meeting it in their individual contexts: by offering access to PCs and skills training to groups whose opportunities had previously been limited (for example amongst community-based staff); by expanding access to electronic resources outside working hours and outside the workplace; or by providing tailored information sources with a local focus.

    Using the right keys to the right doors at the right time

    As stated earlier, one of the aims of the VIVOS project was to involve the site librarians in the research process and to nurture their research skills. As part of this process, a post-project workshop was held to invite feedback from the site librarians and to investigate project-management issues related to the development of these projects. From the discussions at this workshop and from the results of the individual site evaluations, researchers identified several factors that impacted on the successful practical roll-out of the projects:

    The 'vision' or goal

    One of the critical success factors identified at the workshop was the need for the project team to have a clear goal. Although flexibility of approach is necessary to cope with the unpredictable nature of projects that require participation from a variety of stakeholders, there must be an identifiable and attainable long-term 'superordinate' goal (Pinto et al., 1993) or 'vision'. VIVOS set out to establish the extent to which these goals were being attained, to assess levels of user-satisfaction, and to investigate whether the skills and information acquired are being actively channelled into healthcare practice.


    The healthcare professionals surveyed displayed a high degree of motivation in the use of electronic resources and for the development of information skills. Encouragingly, this motivation was noted across the spectrum of job-types and working environments and was seen in enthusiasm to attend training courses and in the fact that several interviewees were keen to develop their skills outside working hours. The challenge here is to harness this initial enthusiasm and keep it stimulated whilst at the same time responding to the needs of those participants who are filled with apprehension at the prospect of using a computer. Maintaining the momentum of enthusiasm can be difficult when librarians are faced with complications beyond their control such as the delayed roll-out of IT equipment at practices.

    Sensitivity to local issues

    It is important to understand local factors that will impact on the take-up of services or training programmes. Projects at the VIVOS sites had to be flexible, responsive to the existing working environments and sensitive to local need. In Cornwall, training was delivered at individual primary-care practices to overcome the difficulties of travel in a rural county and the content of each session was pitched to participants' level of experience. In South Humber, the editorial board of !evidence matters! use evidence-based resources, the monitoring of media coverage, and feedback from the librarian at the Health Authority to identify 'hot topics' with local relevance to appear in this regular bulletin.

    Wide-ranging support

    Virtual outreach projects cannot exist in isolation and benefit from the support of an institutional culture that encourages staff development and recognises that the library has a role in the 'lifelong learning agenda' - i.e. that the library contributes to 'education' rather than simply supplying training courses. In the early stages of setting up a project it is important to have support from all stakeholders including from other departments whose continuing collaboration will be necessary, for example the IT department. Responsibilities must be clear, with service level agreements set up if appropriate. Continuing managerial support is invaluable as the project matures, for example when several members of the same primary-care practice will be absent to attend a three-day training programme, or to ensure the regular supply of updated information to a directory resource like the Pink Book. Library staff will need additional support, including those staff members not directly involved in the project itself. Their awareness and training needs should be identified and met to prevent enthusiasm waning.

    Meeting the needs of users

    Healthcare professionals have different levels of experience in IT, and have different expectations and needs. It is therefore difficult, if not impossible, to provide information sources and/or training that will consistently meet the needs of all users. Evidence was found across the projects to illustrate this and for those sites where the evaluation focussed on training programmes it was a particular problem. Although every effort was made in Cornwall to assess the skills-levels of participants before the sessions, attendance was unpredictable and the trainer had to adapt the course content to suit the majority of participants on the day. At Salford and Trafford the decision was taken by the e-STABLISH team to organise groups by primary-care practice and to represent a variety of roles from each practice. The groups attending the course were therefore not streamed according to previous IT and research experience or interests. This approach does have advantages, and several interviewees specifically mentioned that they enjoyed the chance to learn from the experience of their colleagues, however others found that the range of skills-levels was too wide and that the pace of training left them either bored or unable to keep up.

    Awareness and early identification of benefits

    Although there was widespread appreciation of the skills and support provided by librarians, it was felt that their services could be more widely advertised. Take-up of new services and training opportunities could be improved by a more proactive approach to alerting people to the potential benefits. Gaining confidence with new technologies does require considerable personal effort and investment in terms of time and motivation. The tangible benefits of making this investment need to be reinforced from an early stage. To encourage engagement, librarians need to capitalise on existing motivation and use a wide range of communication methods to promote services. Interestingly, one of the most effective methods of promoting services appeared to be the recommendation of satisfied users who went back to units and enthused about library services to colleagues.

    Keeping the doors open

    Putting projects into practice requires significant effort and collaboration with other professionals and departments. However, a further, and perhaps greater, challenge lies in the maintenance and development of services once their initial impetus has passed. The financial assistance given to kick-start a project may be readily available in the early stages, but the burden of keeping it going or refining it to meet changing needs may fall on over-stretched library resources. Feedback at the post-project workshop contributed to the identification of steps that can be taken to promote the continued success of projects.

    Process of review and feedback

    Ideally, a structured process including the stages of testing, evaluation, taking stock and moving on should be followed. This is not always achievable, but identification of factors that signal success and the appraisal of services against these factors, along with monitoring of user needs and readiness to respond to the changing environment all help to maintain the momentum of projects. The VIVOS evaluation contributed to the review process of the participating projects, and the site librarians commented that added value came from involvement in a wider study that raised the profile, and hence awareness, of their projects at local and regional levels.

    Continuing support for users

    During the VIVOS interviews, participants revealed that their information-seeking behaviours are fuelled by a combination of: educational requirements, patient care, research, the preparation of guidelines and protocols, and audit procedures. Although motivation for the use of electronic resources was found to be high, simply providing services and training is not enough to stimulate independent information-seeking. The majority of interviewees confirmed that they were most likely to look for information when it was required to satisfy a specific need, for example to complete a coursework assignment. 

    In many cases, lack of independent exploration of resources was related to limited time for searching, to logistical issues such as poor availability of fully-functioning PCs, or to weak confidence in IT skills. This can lead to very spasmodic use of electronic resources, which in turn compounds the problem as users forget the skills they have acquired and lose yet more confidence. Post-training 'top-up sessions' provided in Cornwall proved popular, were well-attended and had the advantage of permitting trainees to return with specific questions once they had had opportunity to practise their skills in their daily work environment. An added advantage of the sessions was that new trainees turned up having heard about them by word-of-mouth.

    As they attempt to cope on a daily basis with pressures of time and the demands of patient care, users expressed a preference for the convenience of support as and when they need it, for example via a dedicated help-desk. This can place an unpredictable burden on library staff and resources, especially with increasing numbers of users accessing databases from home and outside conventional working hours. Librarians can help to keep the doors to information open around the clock by providing regular training sessions to ensure that health professionals have the skills necessary for successful independent information seeking. 

    Cascading of skills

    Cascading can be used to widen the impact of a project and give it life beyond the dedicated training sessions. At the Salford and Trafford site participants were expected to cascade skills learnt in training to other staff from their units. In practice cascading was not as extensive as had been intended by the e-STABLISH team although some was successfully achieved. The reasons given at Salford and Trafford and also in Cornwall for not cascading included: low levels of confidence, lack of time, and logistical problems relating to a lack of IT equipment. The lessons learned from this indicate that if trainers do intend a cascading process to take place they should monitor it and perhaps actively pump prime it in the initial stage. Continued support for those cascading the skills would also be a benefit.

    Continued evolution of the project

    Virtual outreach projects are not static and need to respond to the changing environment in which they operate. This can require significant investment of time and effort as library staff build on the results of their review and evaluation processes to shape projects to meet the evolving needs of users. Training sessions must keep pace as new sources of information become available and as participants' skills levels and interests develop. Topical bulletins such as !evidence matters! must retain their focus and individuality to ensure that they are easily identifiable in the midst of all the other information that bombards practitioners and are not simply 'added to the pile'. The transfer of the Pink Book to a Web-based format should make the maintenance process easier but brings with it a new set of problems in terms of who should have access to make changes and how its content should marry with that available through the Trust Intranet. 


    The healthcare practitioners surveyed by the VIVOS project were clearly working under considerable pressure and were at risk of being inundated with information. Outreach services such as those surveyed by VIVOS have an important contribution to make in equipping practitioners with the skills to critically appraise and exploit the resources available, whilst meeting the challenge of guiding users through 'doors' leading to timely, relevant and topical information with a sound Evidence base.

    However, the initial start-up and continued provision of these services can require significant investment of time and resources to ensure: 

  • Seamless and structured training
  • Continuous support for users
  • Constant monitoring of user needs
  • Maintenance of content for topicality, timeliness and relevance
  • Active promotion of services and training programmes
  • Adequate IT infrastructure to support the electronic information resources
  • Successful evolution of the project.
  • As the benefits of outreach services become more apparent and necessary, it is hoped that the shared experience of the sites in the VIVOS study will be of use to other libraries seeking to set up similar services. There was strong support amongst the participating healthcare practitioners for services that can meet their information needs in a way that is flexible enough to fit in with their daily working practices and patterns. Many were reaping the benefits of increased skills and access and had found involvement in the outreach projects to be an empowering and rewarding experience, as is illustrated by this comment from a nurse in Leicester:

    'And for me it was such a great achievement, to roll all that together just seemed like something that I would never ever, ever be able to do......Because I am a doer, I am not a writer or a thinker, I am a hands on person really, so it was a great achievement really.'


    - Abad-Garcia, M., Gonzalez-Teruel, A. and Sanjuan-Nebot, (1999) Information needs of physicians at the University Clinic Hospital in Valencia, Spain. In L. T.D. Wilson and D.K. Allen, (eds). Exploring the Contexts of Information Behaviour. Proceedings of the Second Internation Conference on Research in Information Needs Seeking and Use in Different Contexts. Sheffield, 13-15 August 1998. London: Taylor Graham. pp. 209-225.
    - Chell, E. (1998) Critical incident technique. In G. Symon and C. Cassell. (eds). Qualitative methods and analysis in organisational research: a practical guide. London: Sage, pp. 51-72.Davies R, Urquhart C J, Smith J, Hepworth J B. (1997) Establishing the Value of Information to Nursing Continuing Education. BLRIC Research Report 44. Available from Boston Spa: BLDSC (Thesis Division).
    - Davies R, Urquhart C J, Smith J, Hepworth J B. (1997) Establishing the Value of Information to Nursing Continuing Education. BLRIC Research Report 44. Available from Boston Spa: BLDSC (Thesis Division).
    - Farrell, L. et al. (1999) Virtual evidence: helping primary care practitioners access and implement evidence-based information. Health Informatics Journal, 5 188-192
    - Hunter, J. and Lockyer, M. (1999) Electronic Pink Book: an integrated clinical information system. Guidelines in Practice, 2, 33-35.
    - Hunter, J., Hunter, R. and O'Reilly, D. (1998) By the book: a case study of electronic information integration. British Journal of Health Care Management, 4, 11, 545-548.
    - Pinto, MB, Pinto, JK and Prescott, JE. (1993) Antecedents and consequences of project team cross-functional co-operation. Management Science 39 (10) 1281-1297.
    - Trent FUTURES Project (1999) Annual Report April 1998-March 1999.
    - Urquhart C, Davies R. (1997) EVINCE: the value of information in developing nursing knowledge and competence. Health Libraries Review 14: 61-72.
    - Urquhart C, Massiter C, Thomas R, Sharp S, Smith J. (1999) Getting information to vocational trainees: Report of the GIVTS Project. Library and Information Commission Research Report 206. Available online at http://www.dil.aber.ac.uk/dils/Research/RFocus5/5sub1/Resource/GIVTS0.htm