Assessing the efficacy of EBM teaching in a clinical setting
Gardois P.*, Grillo G**., Lingua C*., Jourdan A.***, Fronteddu S*., Piga A.*
Turin – Italy
* Department of Pediatrics – University of Turin
**S. Giovanni Battista Hospital, Turin
***Medical central library. University of Turin
Background: evaluating IR and EBM skills
Growing popularity of EBM practice   Growing number of EBM courses
Many methods for teaching and learning evaluation
But... only few existing data on the application of the EBM skills in the clinical practice, mainly when:
the working environment is not EBM-oriented
there is an interest in long-term results

Objectives
To assess the long term effect of an educational program for health professionals, based mainly on information retrieval and EBM principles
Secondary goals
•	to understand the role of EBM methods and instruments in clinical practice
•	to assess users’ perception of the main barriers to EBM practice

Our educational program
•	Institution: Pediatrics and Gynecology Departments of Turin University (Italy)
•	Number of courses: 6 (5 with CME credits).
o	4 focused on medical information retrieval:
•	theory of databases
•	basics of information retrieval
•	basic and advanced use of medical databases and search engines
•	bibliography formatting software
o	2 specifically EBM oriented:
•	advanced IR skills
•	basic statistics
•	study design
•	critical appraisal
•	Time span: 2001-03 to 2003-06
•	Average duration: 16.5 hours (min 11, max 28)
•	7 teachers and 6 tutors (librarians, clinicians, statisticians, epidemiologists)
•	126 unique users (140 total); categories:
o	MDs, postgraduating doctors
o	Nurses, midwives
o	PhDs (chemistry, biology, …)
o	Other technical staff
•	average learners/facilitators rate: 3.9 (min 2.7 max 5.3)
Methods: 1. IR skills durability assessment
The multiple choice questionnaire administered after the course was administered again in may 2004 (11 to 38 months after)
 assessment of the Delta (Δ) between the two results
Methods: 2. EBM skills durability assessment
The first part of the Fresno Test (validated in 2002; see Ramos – BMJ 2003) was administered to test EBM skills in 4 areas: clinical question, selection of bibliographic sources, selection of study design, MEDLINE search strategy.

Methods: 3. EBM practice assessment
Qualitative questionnaire with the following main categories of items:
o	Ideal and real percentage of clinical activity devoted to EBM
o	Relative frequency of use of different bibliographic sources
o	Experience of barriers and facilitators for EBM practice

Methods: users invitation and questionnaires completion
Users approached by letter and – after 1 week – by e-mail, explaining the study purpose and methods
Up to 5 phone calls during 3 weeks to make an appointment for questionnaire completion
Questionnaires administered individually in the library, without the use of computers.
45 minutes for completion
Methods: statistical analysis
Software used: StatSoft Statistica 6.1 for MS Windows
Results
1 Basic statistics
1.1 Response rate
1.2 Demographics (gender, age, role)
2. Descriptive statistics
2.1 Δ first – second multiple choice questionnaire
2.2 Fresno test results
2.3 Average responses for any variable in the EBM practice questionnaire
3 Correlations between:
1.2 + 2.3 vs 2.1 + 2.2

Basic statistics: response rate
70 / 126 users  55,6% response rate
N = 70
Based on reported opinions the questionnaire was considered “difficult”.
A higher response rate might have been reached by accepting “remotely completed questionnaires”, but the data reliability would have been lower
Demographics
•	Gender: F 71%, M 29%
•	Age: Mean = 41,03, StdDv = 8,70, Max = 62, Min = 27
•	Role: 72% doctors, 28% other roles
Test results
Delta between 1st and 2nd completion of multiple choice learning evaluation questionnaire: delta%:  N = 70; Mean = -19,43; StdDv = 19,02; Max = 16,67; Min = -73,91

The main barriers affecting the results of the 2nd completion of the multiple choice questionnaire are:
•	Finding contradictory results in the literature
•	Knowledge of English
•	Insufficient number of PCs

The main barriers affecting the Δ pct between 1st and 2nd completion of the multiple choice questionnaire is insufficient access to PCs

Results: Fresno test 
The average total score was 54,99 (valid N 46, Std.Dev. 24,24).

The main barriers affecting the Fresno test total score are:
o	Lack of institutional support
o	Lack of statistics skills
o	Insufficient search strategy building abilities
o	Poor knowledge of English language

EBM practice ideal and real
With a valid N of 66, the average pct of time that our professional would devote to EBM practice was 23,06 (20,10 Std.dev), the real pct of time they devote is 6,29 (7,56 std.dev).

The real and ideal practice of EBM are correlated. Users who consider more important EBM practice tend to have a higher percentage of real EBM practice
Time is an important barrier as to practice EBM skills. There is a significant correlation between the two variables

First 4 bibliographic sources searched (pct. of total weekly research hours) – valid N: 67
 	
•	MEDLINE	31,00	(25,78 std.dev)
•	Proceedings	16,40	(18,22 std.dev)
•	Journals	16,30	(13,38 std.dev)
•	Colleagues	10,07	(16,62 std.dev)

Main barriers to EB practice
AVG points on 1-5 Likert scale: “totally disagree” to “totally agree”
 	
•	Time				3,4	(1,2 Std.Dev.)
•	Isolation				3,2	(1,1 Std.Dev.)
•	Lack of institutional support	3,2	(1,3 Std.Dev.)

Main facilitators for EB practice
AVG points on 1-5 Likert scale: “totally disagree” to “totally agree”
 	
Time	4,0	(0,8 Std.Dev.)
Critical appraisal skills	4,0	(0,9 Std.Dev.)

Other remarks
Users who had good 1st questionnaire results had a greater probability to have also good 2nd questionnaire results
Users who had good 2nd questionnaire results had a greater probability to have also good Fresno test total score results
 Good internal coherence of the assessment methods of the study

Conclusions
EBM skills tend to be lost after some time.
Our users tend to consider ideal EBM practice very important, but the real practice of EBM is significantly lower than the ideal one.
The main barriers to EBM practice, and the more related to the results of our tests are: time, access to PCs, isolation and lack of institutional support.

Further research 
We need larger, multicenter studies to reach more reliable and relevant results
We need to test if – for the practicing professional – regular recalls of EBM concepts are useful
We need to study more extensively the influence of barriers vs EBM practice and to take consequent action.
Bibliography
	(1) 	Closs SJ, Bryar RM. The BARRIERS scale: does it 'fit' the current NHS research culture?... including commentary by Scott E. - Nt Research 2001; 6(5):853-866.
	(2) 	Dunn V, Crichton N, Roe B, Seers K, Williams K. Using research for practice: a UK experience of the BARRIERS Scale. J Adv Nurs 1997; 26(6):1203-1210.
	(3) 	Euler U, Juche A, Bruggenjurgen B, Kunz R, Willich SN. The implementation of evidence-based medicine in clinical practice - External Offers to search for and critically assess medical literature. [German]. - Zeitschrift fur Arztliche Fortbildung und Qualitatssicherung 2002; . 96(5):325-331.
	(4) 	Forsetlund L, Bjorndal A. Identifying barriers to the use of research faced by public health physicians in Norway and developing an intervention to reduce them. J Health Serv Res Policy 2002; 7(1):10-18.
	(5) 	Fritsche L, Greenhalgh T, Falck-Ytter Y, Neumayer HH, Kunz R. Do short courses in evidence based medicine improve knowledge and skills? Validation of Berlin questionnaire and before and after study of courses in evidence based medicine. BMJ 2002; 325(7376):1338-1341.
	(6) 	Ghali WA, Saitz R, Eskew AH, Gupta M, Quan H, Hershman WY. Successful teaching in evidence-based medicine. Med Educ 2000; 34(1):18-22.
	(7) 	Grad R, Macaulay AC, Warner M. Teaching evidence-based medical care: description and evaluation. Fam Med 2001; 33(8):602-606.
	(8) 	Griffiths P, Riddington L. Nurses' use of computer databases to identify evidence for practice--a cross-sectional questionnaire survey in a UK hospital. Health Info Libr J 2001; 18(1):2-9.
	(9) 	McInnes E, Harvey G, Duff L, Fennessy G, Seers K, Clark E. Implementing evidence-based practice in clinical situations. Nurs Stand 2001; 15(41):40-44.
	(10) 	Norman GR, Shannon SI. Effectiveness of instruction in critical appraisal (evidence-based medicine) skills: a critical appraisal. CMAJ 1998; 158(2):177-181.
	(11) 	Parahoo K, McCaughan EM. Research utilization among medical and surgical nurses: a comparison of their self reports and perceptions of barriers and facilitators. J Nurs Manag 2001; 9(1):21-30.
	(12) 	Ramos KD, Schafer S, Tracz SM. Validation of the Fresno test of competence in evidence based medicine. BMJ 2003; 326(7384):319-321.
	(13) 	Scott I, Heyworth R, Fairweather P. The use of evidence-based medicine in the practice of consultant physicians: Results of a questionnaire survey. - Australian & New Zealand Journal of Medicine 2000; . 30(3):319-326.
	(14) 	Smith CA, Ganschow PS, Reilly BM, Evans AT, McNutt RA, Osei A et al. Teaching residents evidence-based medicine skills: a controlled trial of effectiveness and assessment of durability. J Gen Intern Med 2000; 15(10):710-715.
	(15) 	Taylor R, Reeves B, Mears R, Keast J, Binns S, Ewings P et al. Development and validation of a questionnaire to evaluate the effectiveness of evidence-based practice teaching. Med Educ 2001; 35(6):544-547