Evaluating electronic information resources for the NHS in London and the South-East (UK)
Barbara Cumbers (KA24 Project Manager) and Christine Urquhart (Department of Information Studies, University of Wales Aberystwyth)

Introduction
KA24 is a service giving electronic access to a selection of databases and journals to all health and social care staff in London and the South East, regardless of profession or workplace. Previously, staff in social care, the voluntary sector and hospices were not able to access services available to NHS staff, and some NHS staff, notably those in the primary care and mental health sectors, had their access restricted by location. KA24 has a much wider remit, and complements the services available to professionals and the public through the National electronic Library for Health, then in its early pilot stage. As one of the aims of the service was to make provision of information services more equitable, training and support was intended to focus on the groups previously less well served. Accordingly, the evaluation objectives reflected the mission of providing a more equitable service. Evaluation was both quantitative, via usage statistics, and qualitative. 

Aims and objectives
This paper describes how the quantitative and qualitative evaluations were undertaken, and how the findings were synthesised to inform future service plans. The aim of the  evaluation was to examine patterns of usage, to explore reasons for high and low usage, and to examine how the benefits of use relate to policy objectives for clinical governance, continuing professional education and quality of patient care.

The objectives were to:
•	re-assess the enablers and barriers to KA24 use 
•	assess the impact of the service on patient care 
Methods – quantitative surveys
Statistics of users and usage, by profession, by organisation and by individual (user name/password) were analysed. User statistics were analysed as a percentage of workforce wherever possible, to enable realistic comparison between sectors. Similarly, usage statistics could be compared between sectors and between professions by analysis in proportion to users. 

An online survey was conducted in October 2002, and this, plus the regular statistics on usage, formed the basis of the First Year Evaluation report published in March 2003. Despite early success in attracting new users, it became clear that there might be some underlying problems in extending the service to new user groups, and an online survey conducted in May 2003 provided the basis of the qualitative evaluation. 

Methods – qualitative survey
A qualitative survey was conducted during the summer of 2003 by a small team (Jane Durbin and Christine Urquhart) at the Department of Information Studies. The evaluation was done independently, but the team consulted with the KA24 project team regularly. The agreed plan involved:
•	interviews with 30 daily users of KA24, 10 first time users,10 infrequent and less experienced users (to be drawn from around 100 names - 40 acute hospital staff, 40 primary care and 20 mental health/specialist trust) (Target 50 interviews). A group of 10 non-users would be included for interview if these could be identified.
•	questionnaire to be sent to a group including those not registered for KA24 (700 in sample, drawn from all the workforce development confederations covered by KA24, and including a range of types of trust for each).
•	interviews with librarians (5-10) responsible for training and publicity
The interview schedule was intended to cover the main points requested for the evaluation:
•	publicity and registration – is this an easy process? (Question 1)
•	expectations – is the target marketing and promotion correct? (Question 2)
•	experience of use – where is additional support required and are there priorities for usability improvements? (Question 3)
•	benefits realisation – just how does the service benefit patient care and clinical governance, and what are the barriers? (Question 4)
•	working patterns –is KA24 fitting into work practice/routines – whom else do users ask, and where else do users look? (Question 5)
•	training and support – does this appear effective? (Question 6)
•	perceptions of confidence and competence – do some users value additional support or mediated searching? (Question 7)

The questionnaire was intended to complement the interviews, by examining:
•	reasons for registering but not using the service
•	reasons for not registering
•	good and bad experiences of using the service
•	support required.

In the online survey conducted in May 2003, respondents were asked if they were willing to be contacted for interview, and if so, to provide contact details. The sampling was essentially a convenience sample, although care was taken to include a range of sites, and type of occupational role. Non-users were identified in two groups: those who had registered as users but whose user names had never been used (selected at random from the usage data); and those who had never registered (selected at random from NHS e-mail data, and then by exclusion from the user data). Selection was done by the KA24 team, and permission for the release of personal data to the evaluation team was obtained from each individual. 

Interviewees were contacted by telephone and a time for interview was arranged. Interviews were recorded, with the interviewee’s permission, and the interview transcribed. 

The number of interviews obtained was 65 in total (56 for users, 9 for library managers) (Table 1). Response rate for the questionnaire survey was 42.8% (296/700).

Category of user	Totals	Daily user	Infrequent	First time	Non-users	Library managers	
Consultant	5	2	1	1	1		
GP	1	0	1	0	0		
Practice Nurse	0	0	0	0	0		
Junior doctor	7	4	2	1	0		
Hospital Nurse or Midwife	11	2	3	5	1		
Community Nurse or Midwife	1	0	0	1	0		
Allied Professional	10	6	3	0	1		
Scientific or Technical 	4	2	0	1	1		
Researcher	2	0	0	0	2		
Manager/admin	4	2	0	1	1		
Information Professional	20	11	0	0	0	9	
Other 							
TOTAL 

Acute                     n=36
Primary Care (PC) n=11
Mental Health (MH) n=6
Other                      n=3
(including
Ambulance, Social Services, HE-related)	65	29

Acute n=20
PC n=4
MH n=3,
Other n=2,	10

Acute n=7
PC n=3	10

Acute n=5
PC n=2, MH n=2
Other n=1	7

Acute n=4,
PC n=2, MH n=1	9	
Table 1 Distribution of interviewees by role and type of usage

Category of user	Number received	Percentage of total (%)	
Consultant	17	5.8	
Junior doctor	24	8.1	
Hospital Nurse or Midwife	74	25.0	
Allied Professional	7	2.4	
Pharmacist	63	21.3	
Manager/admin	17	5.8	
GP	29	9.8	
Practice Nurse	6	2.0	
Community Nurse or Midwife	6	2.0	
Scientific or Technical 	5	1.7	
Researcher	18	6.1	
Information Professional	12	3.0	
Student nurse/medical student 	13	4.4	
Clinical attachment/observer	3	1.0	
Other unspecified	2	0.7	
TOTAL 	296		
Table 2 Distribution of questionnaires by role
RESULTS

Quantitative

Detailed quantitative data are given in the First Year Evaluation Report (Cumbers, 2003) and in the Twenty Month Evaluation Report (Cumbers, 2004). Overall user and usage data for two years are shown in Figures 1 and 2 respectively. As expected, there is a fairly steady climb in user numbers, with dips occurring at the end of the first year when many user names expired. Usage data are less consistent, but are also as expected from the nature of the service, with peaks occurring in October and November with the start of the academic year, and dips in the summer. The major drop in usage in February 2004 is because the national service (NCC) containing similar resources was also available at the same time. Overall, usage shows a rise over the two years comparable to the rise in user numbers.  

 
Figure 1. Registered users of KA24 for each month from March 2002 to February 2004

Figure 2. Usage sessions of KA24 for each month from March 2002 to February 2004

However, one of the purposes of the evaluation was to see whether KA24 is achieving its aims of reaching previously disadvantaged healthcare staff. To assess this, user and usage data were analysed by sector and by profession. The results by sector are shown in Figures 3 and 4. User figures are for February 2004 and are shown as a percentage of workforce (Department of Health data, September 2002 – the latest available), and usage data are shown as total sessions for the sector over two years divided by users (Feb 2004).

Figure 3. User registration as a percentage of workforce, by sector. Non-NHS staff are included in other users

Figure 4. Usage sessions per registered user, by sector. Non-NHS staff are included in other users

Proportionally fewer staff are registered to use KA24 in the mental health and primary care sectors, and they use it less, than staff in the acute sector.

Some professions, notably nurses and allied health professionals, were also previously disadvantaged in information provision. Realistic analysis of user registrations by profession was not possible because workforce data were not available in sufficient detail. However, analysis of usage sessions per registered user by profession (Figure 5) is possible, and shows that doctors use KA24 less than the previously disadvantaged groups. The lower usage by managers was expected because KA24 contains little material specifically for this group.

Figure 5. Usage sessions per registered user, by profession. Other usage has been excluded from this graph because it includes usage in libraries and by information professionals and is therefore disproportionately high

One of the quantitative evaluation criteria set by the KA24 Project Board was for each trust to reach the average for its sector in the previous quarter in two values: the percentage of the workforce registered; and the percentage of registered users who are active users of the system. The idea was to target the under-average trusts in publicity and training, and assumed that other variables were similar for all trusts within the same sector. Data were cumulated for each quarter, and an average value for each trust obtained. This value was then compared to the cumulated figure for the next quarter. The percentages of trusts that achieved the targets are shown in Figure 6. 

Figure 6. Percentage of trusts that achieved the target of the average for their sector in the previous quarter, in percentage of workforce registered (left) and in percentage of registered users who were active users (right)

The success rate was low (about 50%), mostly as a result of the unexpectedly high spread of values within sectors that put the average for that sector out of the  achievable range of many trusts. Non-rigorous investigation revealed a loose direct correlation between trusts with high investment in libraries and those that achieved high registration and usage values.

Qualitative

The 738 online survey responses that indicated willingness to participate in interviews were grouped into daily users, infrequent users (using KA24 less than once a month), middling users (using KA24 on a variable basis, somewhere between a couple of times a week to once a month), and first time users. Comparison of the subgroup with the entire group (Table 4), indicated that the subgroup pattern of purposes was similar to that of the total group, with the exception of research purposes (proportionally smaller in the subgroup) and the mixed purposes (including patient care) (proportionally higher in the subgroup). The staff profile of the sub-group is also very similar to that of the whole group. The subgroup seems, therefore, to reflect the whole group in the main baseline characteristics. 

The interview findings follow, divided by theme.

Reason for using KA24	Grand total n=3709
(%)	Totals
n=738
(%)	Daily users
n=48
(%)	Variable and occasional users n=479
(%)	Infrequent users
(less than once a month) n=57 (%)	First time users n=154
	
Directly related to patient care	509
(13.7%)	103 
(14.0%)	15 
(31.3%)	78 
(16.3%)	10 
(17.5%)	0	
Research	715
(19.3%)	117 
(15.9%)	5
(10.4%)	101
(21.1%)	10
(17.5%)	1	
Coursework	566
(15.3%)	101 
(13.7%)	7
(14.6%)	82
(17.1%)	12
(21.1%)	0	
CPD	217
(5.9%)	46 
(6.2%)	4
(8.3%)	30
(6.3%)	12
(21.1%)	0	
More than one reason, including patient care	688
(18.5%)	180 
(24.4%)	10
(20.8%)	160
(33.4%)	10
(17.5%)	0	
More than one reason, not including patient care	133
(3.6%)	21
(2.8%) 	3
(6.3%)	17
(3.6%)	1
(1.8%)	0	
Other	66
(1.8%)	14 
(1.9%)	3
(6.3%)	10
(2.1%)	1
(1.8%)	0	
No answer	815
(22.0%)	156
(21.1%)	1
(2.1%)	1
(0.2%)	1
(1.8%)	153
(99.3%)	
Table 3 Profile of reasons for using KA24
Profile of daily users
Of the 48 daily users, 52.1% needed the information when they last used the service for a reason which included patient care (Table 4). Most (83.3%) had obtained the information they wanted, and most (91.7%) find the KA24 service very useful. A lower percentage find the service very easy to use (64.6%), with 33.3% finding the service fairly easy to use. 

Profile of infrequent users
Of the 57 infrequent users, the reasons for using KA24 were spread across patient care, research, and formal education with 35.0% of the searches related to patient care reasons (Table 4). Over half (54.4%) had found the information they wanted, 26.3% found something useful, but nearly one fifth (19.3%) did not find anything useful. Unsurprisingly they were less enthusiastic about the service than the daily users, with only 49.1% claiming it was very useful, and 43.9% assessing it to be fairly useful. However, none claimed it was not useful. Around a quarter (22.8%) find KA24 very easy to use, 56.1% find it fairly easy to use, and 19.3% find it not very easy to use, and 1.8% reported it was difficult.

Profile of the ‘middling’ users
The majority of these users have used KA24 more than four times, around two or three times a month, on average. Of these 479 users, 49.7% had wanted information for reasons which included patient care, 21.1% for research, 17.1% for coursework and 6.3% for updating and CPD (Table 4). Most (72.0%) had obtained the information they wanted, 22.1% had at least found something useable and only 5.4% had not found anything useful. The majority (79.3%) find the service very useful, 18.4% find it fairly useful and only 1.0% do not find it very useful. In this group of users, there are fewer (proportionally) who find the service very easy to use (36.3%) against 58.2% who find it fairly easy to use and 3.5% who do not find it very easy to use.

Profile of the first time users
These users answered only some of the questions. Most (34.4%) were searching from home, 31.2% were searching from their own desk at work, 16.2% from elsewhere at work.

Overall profile
The online survey respondents (n=3709) were dominated by the more experienced user group with 67.8% of responses coming from experienced users (had used KA24 five or more times), 11.9% from less frequent users, 19.4% from first time users (and 0.9% provided no answer to this question). 

Educational benefits
Interviews revealed that the big advantage of KA24 is access to full text journals, rated higher than access to databases by the majority of health professionals. The daily user group divided into three groups: the librarians, the drugs information professionals/pharmacists and the clinical staff. Of these:

•	Clinical staff stressed the attraction of access to full text journals (12 mentions, 1 mentioned databases as well)

•	Drugs information staff stressed the importance of EMBASE access (6 mentions, with 1 mentioning full text journals as of equal importance).
‘Definitely and the main thing is the access to EMBASE, and to use the OVID interface....one thing we have found very helpful is the access to the full text journals.’ [pharmacist]
•	Librarians were more likely to view the advantages of more databases and full text equally (7 expressed no particular preference, 1 mentioned full-text as the main attraction and 1 mentioned access to more databases as the main attraction.

Although KA24 started off as a database service, with access to full-text journals as an add-on, it seems that for the clinical community, core content to them would mean access to full text journals, not databases.
Clinical impact
Benefits to clinical practice are being realised. 
‘Yes to the point sometimes I will leave the patient for a minute or two to access a certain article and come back and talk to parents, that is the way life is changing in the hospital and patient care is evolving.’ [doctor]
‘You can access for study or the latest literature on it very quickly, even whilst the patient’s with you or maybe whilst they’re getting changed for treatment.’ [allied health]
Information obtained from KA24 underpins clinical decisions, both current and future.
‘What I find gets discussed in our management decisions.’ [doctor]
‘We’ve got a couple of research groups here and each person will take a topic and investigate it...I think just being able to do from the computer’s actually made a huge difference...to be able to collect that stuff immediately.’ [junior doctor]
A few users (junior doctors in particular) have a very clear idea of their ideal search environment.
‘It would be nice also to have the different databases that are accessible through KA24 interlinked with each other so you can get a reference if they are full text available from another source, but I’m not sure if that’s possible, like for example through Proquest you have access to full text journals and if there’s a link with OVID, so if you’re searching OVID then you get a hint that full text is available through Proquest.’ [junior doctor]
Promotion
Library staff appeared to be informing potential users of the service, and information sheets in the library are also used. Supplemental advertising such as flyers, or posters were mentioned (though rarely) – there may be a subconscious registering of information in a payslip but it seems likely that general campaigns such as putting information in payslips has to be accompanied by more direct promotion. Most libraries saw only one new user a day, which implies that clinical champions or contacts need to be used to persuade staff to use the service. Examples of enthusiast departments identified in the survey include:
•	practice development units
•	clinical audit
•	research projects (including systematic review projects) / research nurses
•	clinical governance
•	medicines information/pharmacies
Tracking service usage
Registration poses a few problems. The qualitative survey found that registration problems and delays were averted by the fairly common practice of password sharing, when registered users allow other staff access to the service on the registered user’s password. 
Training and support
Those who had received training from the library had found this useful, but a large proportion of the interviewees did not see the need for formal training, particularly when this is difficult to undertake with the variable nature of their clinical workload. Nearly one in ten of the questionnaire respondents did not see the need for training at all. Around half the questionnaire respondents had received some sort of the training, usually informal training from the library on an ‘as and when’ basis. Interestingly, and probably a reflection of the amount of searching done at home, questionnaire respondents used the online guides within the databases as much as advice from colleagues. Working through tutorials was not as popular as specific advice when required, presumably. Adding up the figures from the questionnaire survey suggests that there is a training gap. If 50% have received some training, 10% are not interested at all, and around 10-20% are making use of online guidance or colleagues’ advice,  then that leaves 20-30% who might appreciate some training.

As one of the main advantages to the users of using KA24 is saving time, over other means of obtaining the same information, several expressed interest in some assistance or advice on ‘shortcuts’ or more advanced features they could use to save time and obtain a better focused search. Libraries might promote ‘help-desk’ support more, as that would give them better insight into the routine problems (and successes) experienced in their locality.

There were many similarities across all the groups (daily users, first time, infrequent, and non-users) in general feelings of competence and confidence in searching. It cannot be assumed, therefore, that a first-time user is a novice searcher, nor that infrequent users see themselves as such.  The number of regular Internet users across the interviewees was startling, and confirmed by the questionnaire data which indicated that nearly three in four of the questionnaire survey respondents used the Internet at least once a week. KA24 is likely, therefore, to be viewed as a search engine, although a special type of search engine.

Nearly two thirds of the questionnaire respondents could access the Internet from home, and nearly half from their own PC at work. A third had access to a shared PC, or a public PC at work. Only 6% reported problems in access to a PC at work, although some staff in mental health trusts reported they had no option but to use the KA24 service from home.

The analysis indicated that it might be more appropriate to target training and support according to the type of activities or purposes, rather than the competence and confidence of the users. For example, several groups of users could be identified from the interview data:
•	A ) Novices, possibly using KA24 to help in coursework, with less confidence using computers (often nurses)
•	B) Internet-savvy, using KA24 for coursework and also some research at work (often allied health professionals)
•	C) Research active/researchers/ audit or practice development, using KA24 on a regular basis for work, likely to have had (or to have) experience of HE facilities 
•	D) Infrequent or occasional users who use KA24 for clinical practice or personal research interest as a supportive activity, and may use other resources (could include GPs)
•	E) Information professionals (including medicines information and library staff)
•	F) Junior doctors (often have good ideas for service enhancements)

Benefits, and problems, of combining quantitative and qualitative evaluation data
The process illuminated the problems of trying to define groups of users. While it is possible that the service usage data can be divided into usage patterns by frequency, the qualitative survey showed that one password did not necessarily correspond to one user. It was difficult to assess the true scale of this practice, as participation in the qualitative survey was voluntary. For training and support, frequency of usage might not provide good clues. The qualitative evaluation did differentiate some other user groups, and help to profile patterns of need, and training needs. The interviews also helped to clarify where service champions might be found.

Cullen (2004) discusses approaches to evaluation of digital libraries in the health sector, concluding that it is necessary to look beyond the positivist scientific model of the randomised controlled trial. Such a controlled trial design would not have been possible with the KA24 evaluation , as the research questions focused on the softer issues as well as the trends in usage. The MLA recommendations are based on the balanced scorecard approach, examining the customer, financial, internal business and organisational learning perspectives. Cullen concludes that a model for the evaluation of digital libraries in the health sector could include measures of timeliness and efficiency, technical and infrastructural perspectives, clinical outcomes, standardisation of data on an agreed basis, and information on benefits and user effort. Experience with KA24 project evaluation suggests that the focus might need to be staged, as the project progressed. The initial quantitative evaluations were concerned, quite naturally, with the efficiency and technical perspectives as well as the clinical impact.  Questions emerged about the promotion and the training and support, and these required a qualitative study. It was not possible to do a full observational ethnographic study, but the interviews did give a better indication of the way KA24 services fitted into the use of the Internet at work and at home for all sorts of purposes. The interviews did not quantify how much time use of the service saved the interviewees over other ways they would have found the information (as was done in an evaluation of the pilot National electronic Library for Health (Urquhart et al. 2001)). Both interviews and questionnaires indicated that service users valued the time savings, however, as a benefit of the service. 

Evaluation is costly, and managers may be tempted to do a partial evaluation, or focus on one or two aspects only. Experience with KA24 suggests that this might be a false economy, as it was only by taking a more holistic approach, covering the various perspectives of the balanced scorecard, that a better picture emerged.
Conclusion
The generalisability of the KA24 evaluation findings is limited by the difficulty of finding interviewees within the time period specified for the evaluation, and the fact that the evaluators had to rely on interviewees’ subjective opinions on clinical benefits, which could not be checked against other records. Future surveys could focus more on the time savings, but these would have to be subjective estimates. The qualitative survey findings complemented the ongoing quantitative evaluation and both clarified some of the enablers and barriers, as well as revealing aspects of service usage that would be hard to reveal with the quantitative data alone. Evaluation should be holistic, measuring effectiveness, efficiency and how the service fits into the user’s work, home and education environments. The KA24 evaluation work was staged, and that worked well, with questions raised from the quantitative evaluation informing the later qualitative evaluation.

REFERENCES

Cullen, R. (2004) Evaluating digital libraries in the health sector. Part 2: measuring impacts and outcomes. Health Information and Libraries Journal 21 (1): 3-13.

Cumbers, B. (2003) KA24 (Knowledge Access 24) first year evaluation report. Report for the KA24 Project Board.  Available from: http://www.hilo.nhs.uk/pdfs/FirstYearEvaluationReport-Public.pdf (Accessed 3 June 2004)

Cumbers, B. (2004) KA24 (Knowledge Access 24) twenty month evaluation report. Report for the KA24 Project Board. Available from: http://www.hilo.nhs.uk/pdfs/TwentyMonthEvaluationReport-Public.pdf (Accessed 3 June 2004)

Durbin, J and Urquhart, C. (2003) Qualitative evaluation of KA24 (Knowledge Access 24). Project report for KA24 Project Board.  Available from: http://www.hilo.nhs.uk/pdfs/QualEvalKA24Report-Public.pdf (Accessed 3 June 2004)

Urquhart, C, Yeoman, A, Cooper, J, Wailoo, A. NeLH pilot evaluation project. Final report to NHS Information Authority, NeLH. Aberystwyth: DILS, 2001 (available via research web pages for the Department of Information Studies, University of Wales Aberystwyth)