Rural but not remote! Access in outback Australia.  Report on the implementation of Personal Digital Assistants (PDAs) for medical students, clinical teaching staff and health librarians at the Rural Clinical Division, School of Medicine, University of Queensland, Australia

Andrew Heath, Lisa Kruesi, Kaye Lasserre, Heather Todd and Sarah Thorning. (University of Queensland Cybrary)

Abstract
Funding to provide 95 Personal Digital Assistants (PDAs) to medical students, located at the Rural Clinical Divisions (RCD) in outback Australian Queensland regions, was obtained in 2003 from the Australian Government Department of Health and Ageing.  The PDA Project commenced in October 2003 and will continue until the end of 2004.   The objectives of the Project were to provide access to authoritative, evidence-based information to rural medical students at their point of need and to address equity issues for students placed in remote or isolated areas with limited access to information and technology infrastructure.  The project will be used to identify how such technology can help to educate and prepare students for practicing medicine in the 21st Century and as a means to identify appropriate resources, training and support required to utilise PDAs.  The Herston Medical Library, University of Queensland, provided a senior Liaison Librarian as the Rural Pack Project Coordinator, along with two Liaison Librarians, who are based at the rural sites in the Central Region and the South West Region of Queensland, each working at the main hospital within each region.  The Coordinator of the Project has led a team of clinicians, librarians, information technology representatives, a medical education officer and medical students to evaluate the technology.  Included on each PDA distributed to PDA Project participants are e-resources, including a major drug index, medical textbooks, a medical dictionary and a medical calculator.  The Microsoft Office Suite of software, including Word, Excel, PowerPoint and Outlook are also included on each PDA.  A report on the implementation of the Project, study design and outcomes reported during 2004 is provided.  Various challenges, such as licensing complexities, that were encountered with the introduction of this innovative service are outlined, as well as future plans.

Background
Personal Digital Assistants (PDAs) have emerged as a popular mobile technology to support the clinical information needs of health professionals and medical students. PDA sales are expected to reach 58 million by 2008.  The PDA Project at the University of Queensland (UQ) was funded by the Australian Government Department of Health and Ageing.   The UQ School of Medicine, in partnership with the Cybrary, undertook the Project to introduce PDAs to staff and students located at the School’s Rural Clinical Division (RCD).  When the Project commenced in 2003 there were no PDA projects underway in any Australian medical schools.

The UQ School of Medicine’s RCD was established in January 2002 with two regions in Queensland, the Central Region and the South West Region based in provincial teaching hospitals in Rockhampton and Toowoomba, respectively. It is part of a national rural education and training network funded through the Australian Commonwealth Government's Rural Health Strategy by the Department of Health and Ageing.  The RCD provides an opportunity for medical students to undertake their clinical training across a network of hospitals, General Practitioner (GP) surgeries and community medical centres in locations throughout Central and South West Queensland.  Whilst the RCD is part of the UQ it resides in Queensland Health (QH) premises and works within the QH information technology environment.  QH is the major State Government body that provides health services for residents of Queensland.

The UQ Cybrary, holds one of the largest collections amongst Australian academic libraries and by far the largest collection in Queensland.  The Cybrary combines physical space and cyberspace in the delivery of real and virtual information resources.  In 2004 funding for the Cybrary from the University budget was AUD$27 million (EUR€15 million). The service comprises 13 branch libraries, 3 in the major Brisbane teaching hospitals, with the largest being the Herston Medical Library (HML).   The Cybrary employs 248 staff that provide library and information services up to 84 hours per week. 

The HML coordinates Cybrary services provided to the RCD.  Liaison Librarians are based in each RCD region and their major role is to: represent the expertise of the Cybrary; optimise the Cybrary investment in online and real health information resources; and work in partnership with the RCD staff and medical students in their Regions.

When the opportunity arose to obtain funding from the Australian Government to develop technology based services for the Rural Clinical Division, a PDA project was viewed as a suitable choice based on successful descriptive reports of usage in the literature.,,   

PDAs were becoming mandatory tools required to support the teaching, learning and the clinical information needs of students in such respected medical schools as Harvard and UCLA. Two primary reasons at UCLA for requiring students to have a PDA are ‘to enable ‘point of contact’ access to information resources; and to prepare students for practicing medicine in the 21st century’” 

At the commencement of the Project there were no studies of PDA technology in either the Australian or the International rural environment for medical students.  To date no Australian projects in a clinical setting have been formally reported but other studies that have been underway, such as the trial of 18 PDAs to 35 junior doctors - also undertaken in Queensland.

An objective of the Project was to optimise the application of contemporary information.  It has been found that there is a greater need for more studies that provide standardized, independent and combined qualitative and quantitative aspects of medical applications for PDAs.  The vast majority of appraisal material that exists is mainly drug reference applications rather than the many other types of medical applications that exist.

A recent study based on a randomised trial design has been one of the first to evaluate the effectiveness of a handheld clinical decision support tool.  It also claims to be “one of only a few trials on evidence based medicine learning in the undergraduate setting…”    The study concludes that “rapid and convenient access to valid and relevant evidence on a portable computing device can improve learning in evidence based medicine, increase current and future use of evidence and boost students’ confidence in clinical decision making.”  

In addition to the provision of access to information resources, various reports highlight the range of ways PDAs are being used, including:
•	Provision of timetables, lecture schedules and contacts 
•	Distribution of study materials, class notes, readings and as a testing environment
•	Recording hospital case log notes and learning portfolios,
•	A communication channel to students for lecturers and course administrators.
Based on the positive experiences reported in the literature many of these applications have also been explored as part of the UQ PDA Project.

The PDA Project
The formal Project objectives were to:
•	Provide access to authoritative, evidence-based information to the RCD medical students at their point of need.
•	Contribute to a positive rural medical education experience and address equity issues for students who are placed in remote or isolated areas and have limited access to information and technology infrastructure.
•	Optimise the application of contemporary information and telecommunications technologies in healthcare and to prepare students for the practicing medicine in 21st Century.
•	Provide a platform for the UQ, School of Medicine to assess the applicability of this technology to the teaching and learning environment.
•	Provide the UQ Cybrary with experience in the provision of appropriate resources, training and support for PDAs.

The Project commenced in July 2003 with a business plan that detailed the management of areas including: key stakeholders, deliverables, budget, schedule, resources, reporting requirements and an evaluation methodology. A pilot phase was undertaken during October to December 2003, followed by a Project phase from February to December 2004. The chief areas of investigation were information resources, administrative applications and teaching and learning applications for PDA use.

The budget for the Project included AUD$72,000 (EUR€41.312) for the hardware (see Table 1 for details) AUD$30,000 (EUR€17.213) for information resources and AUD$16,000 (EUR€9.180) for a part-time Project Coordinator. Evaluation was undertaken at both phases (the pilot and the Project) using a combination of surveys, focus groups and usage data. This paper reports upon the survey results received to date and how project objectives are being met.

The PDA Project was coordinated by a Senior Librarian from the Herston Medical Library, with project management provided by a staff member from the School of Medicine. A Project Reference Group was established comprising 14 members including four clinical lecturers, four librarians, two RCD managers, a QH representative, an information technology expert, a medical education officer and a medical student.  The Reference Group have met eight times to discuss Project matters via teleconference during October 2003 to August 2004. 

Administrative procedures were developed to help ensure the security of the PDAs. An agreement was developed that specified requirements based on UQ Library conditions of use, including loan requirements and overdue penalties, UQ IT policies and QH Information privacy standards.  The latter indicated that recording identifying patient information was not permitted. The agreement listed the components of the PDA package for loan. Each participant was required to sign the agreement prior to receipt of the equipment.

The RCD Librarians undertook the asset management.  The units were barcoded and numbered and lent using the UQ Library Management System.   A detailed inventory was maintained with the location and borrower of the items noted and any changes to the status of the units recorded.  Unused equipment was stored in locked cabinets in the RCD Library Services.

It was not possible under University insurance policies to cover the cost of replacing any lost or damaged equipment because the units were each under the value of AUD$1,000 (EUR€574). Participants were encouraged to list equipment under their residential insurance policy. In the event of any loss or irreparable damage consideration for liability was to be determined by the relevant staff of the School of Medicine.

The pilot phase 
The pilot phase of the PDA Project was undertaken from October to December 2003 during the final clinical rotation of the year. Participants included Year 3 medical students permanently located at the Rural Clinical Division and selected academic and administrative staff.  The aim of introducing the technology to this cohort of students was to gauge the use and suitability of the technology, develop the technical skills required and to determine if the mix of information resources and software was suitable for the Project phase.  A total of 40 units were purchased and 21 of these were distributed to the students. Participating staff, the RCD Librarians and the Project Coordinator were each allocated a unit. Limited distribution of PDAs helped to make the pilot phase manageable. 

A request for tender for the supply of 40 PDAs and 40 128 MB Secure Digital (SD) Memory cards, was issued with the following requirements:
Table 1.
Operating system	Mandatory - Pocket PC 2003 desirable.	
Processor	Mandatory - 200mHz or above	
Main memory	46MB minimum 	
Expansion slots	SDIO (Secure Digital) card slot mandatory 
CF Card Slot desirable	
Docking station (Desktop cradle and charger)	Mandatory	
Infrared Beaming	Mandatory	
Wireless (bluetooth, WiFi)	Desirable	
Additional Memory	Provide separate pricing for 64MB, 128MB and 256MB SD Cards	
Protective cases	Mandatory	
Presentation hardware and software required for training and demonstrations.	3 PDA’s will require direct connection to projectors for presentations. 	

Following trials of various brands of PDAs and the evaluation of tender responses, the Hewlett Packard IPAQ H2210 Pocket PC was selected on the basis that:
•	All specified mandatory requirements were met.  In addition the unit featured a Compact Flash expansion slot and Bluetooth wireless capability. 
•	Processor speed of 400 Mhz was the fastest available Pocket PC processor.
•	The unit was one of the smaller and lighter of all proposed PDAs. 
•	Additional software including a PowerPoint viewer was included.

The Palm and the Pocket PC are the main operating systems available for PDAs. The Pocket PC was chosen by the Reference Group as the Windows platform is the current desktop environment at the UQ. For integration and support purposes it was the preferred model. While Palm had dominated the PDA market and had more clinical information products and applications available, it was clear that all new products being developed were compatible with both operating systems and some only for Pocket PC (for example UpToDate). Wireless communications were not a feature of the Project because QH did not permit wireless communications for security reasons.

The Pocket PC 2003 software provides for basic personal information management functions including calendar, contacts database, to do lists and basic Microsoft Office applications, such as Word, PowerPoint, Outlook and Excel. 

The Project Reference Group considered various information resources. The preferred mix of resources was: a general clinical textbook, a medical dictionary, a clinical calculator, a drug information database and an evidence-based medicine resource. Following consultation with medical students and clinical lecturers, the following suite was agreed upon:

•	Micromedex – This resource included drug information, alternative medicine, acute care and toxicology databases and a drug reaction tool. It was available at no extra charge as part of the Cybrary’s online subscription. (Details are available from: www.micromedex.com/products/mobilemicromedex)
•	Archimedes  – A  free clinical calculator. (Details available from: skyscape.com)
•	Oxford handbook of clinical medicine - (AUD$85/ EUR€49 each)
•	Oxford handbook of clinical specialties - (AUD$85/ EUR€49 each)
•	Oxford concise medical dictionary – (AUD$28/ EUR€16 each)
•	Clinical evidence - An international source published by BMJ Publishing Group.    Provides access to evidence for effective health care, featuring treatment and prevention of nearly 200 medical conditions. This product was available for a free trial during the pilot phase. (Details available from: www.clinicalevidence.com)

In addition the software Adobe Acrobat and the Mobipocket readers were loaded on every PDA to allow viewing of PDF documents and textbooks.

The UQ Cybrary purchased individual licences for the Oxford handbook of clinical medicine, Oxford handbook of clinical specialties and the Oxford concise medical dictionary. Neither site licenses nor discounts were available for these bulk purchases.

It was envisaged that the units would be distributed with the software and information resources fully loaded. Unfortunately a number of challenges during the setup of the PDAs were experienced, including:

Individual registration
Most PDA products are licensed to individual PDA units using some form of unique device ID and also require a form of registration for each unit. Having 40 units to commission with six programs (240 registrations, downloads and installations) proved to be a time consuming and difficult process. 

Installation of Micromedex
Installation of Micromedex proved to be a problem. While the installation of this product on a single unit is relatively straightforward, this did not prove to be the case with the installation onto multiple units. After several consultations and tests with Thomson Healthcare support in the USA, a means of installing the product was reached. This allowed units to be loaded with the five databases, but did not allow for monthly updates to take place. Tests on the QH network revealed that Micromedex could not be installed via the QH network on PCs used by Library staff or students. This was due to security restrictions on the QH network, blocking the sending of information required by the Micromedex site. These problems remained to be fully resolved for the Project phase. As the pilot was relatively short it was not considered crucial for the updates to take place.

Imaging tools
Software imaging tools tested during this time were not found to be useful due to the individual licensing requirements mentioned above.

The initial setup of units, loading software and information resources was undertaken by the Project Coordinator. RCD Librarians located in Rockhampton and Toowoomba provided immediate support and advice to users.  Higher level advice was provided by the Project Coordinator and the Library Technology Service. Additional assistance was also provided by the School of Medicine, IT Section. The repair of breakdowns was undertaken by the suppliers Velonte (formerly IPEX).

Introductory on-site training was provided by the Project Coordinator to all participants.  A user group meeting was convened by the Librarian in Toowoomba as a forum for staff and students to discuss problems and share solutions. 

The pilot evaluation was based on:
•	Suitability of hardware and software
•	Appropriateness of information resources
•	Appropriateness of training materials
•	User support arrangements

A pre-pilot survey was administered prior to the distribution of the PDAs. The survey covered issues such as current knowledge and level of use of various IT functions. This included personal information management tools, basic computing functions (for example word processing), use of information resources and perceptions about the usefulness of PDAs. A post-pilot survey was given at the end of the pilot. 

The suitability of hardware and software was gauged by questions relating to the level of acceptance of the technology by the participants. The post-pilot survey indicated that the majority of respondents (93%) reported that they “learned to use [the PDA] with some assistance and have required minimal support”, with the majority of these not requiring any support.  The amount of use (and hence relevance) was also recorded and survey responses showed that 62% of participants used their PDA at least once a day or more and 79% reported that the PDA was convenient or very convenient to use. 

The survey also asked participants to indicate their assessment of the impact that such technology has on studying medicine in the rural environment. Responses indicated the PDAs:
•	‘would improve my performance as a clinician’ (79% agreed or strongly agreed) 
•	had ‘optimized my access to contemporary information at the point of need’ (86% agreed or strongly agreed)
•	‘helped me to address diagnostic questions effectively (86% agreed or strongly agreed)
•	86% of respondents agreed that ‘access to reference materials on the PDA contributed to [their] educational experience’. 

Responses to specific questions relating to the information resources were as follows:
Table 2. Information Resources
	Not useful	Somewhat useful	Undecided	Useful	Very useful	
The Oxford Concise Medical Dictionary	0%	8%	15%	54%	23%	
Archimedes (Clinical Calculator)	0%	8%	58%	25%	8%	
Micromedex (drug information)	0%	8%	23%	15%	54%	
Clinical evidence (evidence based practice)	0%	0%	54%	31%	15%	
Oxford Handbook of Clinical Medicine	0%	0%	8%	23%	69%	
Oxford Handbook of Clinical Specialities	0%	0%	15%	23%	62%	

Figures detailed in Table 2. indicate the two handbooks were regarded as very useful by over 60% of respondents. The drug information database (Micromedex) was considered useful by 54% of participants. Specific comments received suggested the inclusion of MIMS as an alternative to Micromedex as it contains Australian drug brand names. Clinical Evidence was not well used and it is speculated that this is due to the students’ lack of familiarity with the resource and its limited content. Archimedes, the clinical calculator, was the least useful as the units for some calculations were not suitable in the Australian context.

Users reported that the training was useful or very useful across all topics covered with the exception of synchronization which was not able to be demonstrated during the training sessions.

User support arrangements were reported to be adequate. The majority of participants (71%) did not experience any problems using the PDA. Problems reported were of a minor nature, for example; “it froze several times and had to be reset” and the “drug resource would freeze up”. A log of support issues was maintained during the pilot. 

In addition, the surveys provided a wealth of suggestions for improvements in the utility and functionality of the PDA to be implemented in the Project phase. These included using the PDA to record logbook information and as a means to promulgate the academic calendars.

The pilot provided an excellent proof of concept in the areas of the usefulness of the PDA and the information resources at the point of care, the potential for contributing to an enhanced learning experience for the students and demonstrated their enthusiasm for the Project. In addition a number of challenges were identified and solutions sought where possible. Technical issues including difficulties with resource installation and synchronization, security and firewalls were identified and solutions put in place for the Project phase. Issues with the licensing of resources were also highlighted, in particular licensing only to individual PDAs which were not subsequently interchangeable with new devices. A number of useful suggestions for additional information were received from the students. These were investigated for the Project phase. The absence of site licences for the provision of resources to an entire user population was an added difficulty and expense.  In some cases where the Cybrary already had a license to the online version it was compelled to buy the same content twice. This is the case with the drug database MIMS.

The Project phase
During the Project phase PDAs were distributed to 73 permanent RCD students and the remainder of units went to selected academics and Project Reference Group members.  The trial will finish in December 2004 with a final report to be completed early 2005.  In 2004 an extra 55 PDAs were purchased, making a total of 95 available for the Project phase.

The Reference Group reconsidered the set of information resources to be made available during the Project phase on the basis of feedback received during the pilot phase.  It was recommended that the following be provided for each PDA:
•	The Oxford handbook of clinical medicine
•	The Oxford handbook of clinical specialties
•	The Oxford concise medical dictionary
•	MIMS (An Australian drug database. Cost: AUD$109/ EUR€63 unit)
•	Archimedes clinical calculator
In addition the ClearVue document reader (to allow the viewing of PowerPoint files), Acrobat reader and Mobipocket reader were loaded on each unit.

Resources included in the pilot phase that were not made available during the Project phase were Clinical Evidence and Micromedex. Clinical Evidence had become a subscription service and the cost to licence the resource for 95 PDAs was over AUD$10, 000 (EUR€5.740). The Reference Group considered that the feedback received during the pilot phase did not support the purchase of this product. Micromedex was at the time of the Project phase creating a new version of its PDA software and it was decided to await its release. The new version became available shortly after the distribution the PDAs and was made available to participants if they chose to install it. Efforts were made to identify an alternative medical calculator with Australian units however none was found. It was decided to include the Archimedes calculator to provide access to at least those calculations that were useful. 

Following the experience with the pilot phase it was negotiated with the supplier that the extra 55 Project phase PDAs were delivered with:
•	The battery charged and the device ID, date, time and regional settings configured
•	Required folders created on each PDA, including the relevant UQ documents saved to these folders
•	Customised background image with the UQ Logo
•	ClearVue presentation viewer, Adobe Acrobat Reader, Mobipocket Reader and Archimedes Medical Calculator loaded
•	Contacts and Calendar information for the South West PDAs and Contacts only for the Central Queensland units imported to the PDAs

Guides were written for the configuration of the devices and the installation of software and the information resources to the PDAs.  This was to assist the RCD Librarians who provided frontline support of the PDAs. Backup procedures were developed to ensure that PDAs could be easily restored in the event of failure. This involved a strategy of backing up the contents of each unit to a CD and to the SD card on the unit. This meant a unit could be restored easily by the user or by the RCD Librarian.
 
Ongoing technical support arrangements were negotiated with the Library Technology Services.  Procedures were developed to setup computers in the RCD Library Services for participants to synchronise their PDAs. These were tested on the UQ network and agreement was gained to allow this to happen on the QH network where the RCD Library computers were located.

A Listserv was established to communicate with the participants. The list was used to communicate with users, provide an avenue to seek answers to questions and provide a forum for discussion.  User groups and ‘Clinic Days’ were held where participants could drop in to ask questions or have programs re-installed.  This was an attempt by RCD Librarians to manage their time and minimise interruptions to work flow.  Generally most troubleshooting took place outside these more formalised structures.   

Training was provided for participants in the Project phase during April 2004.  In Rockhampton two training sessions each with ten participants were conducted.  In Bundaberg two training sessions were held each with six participants and in Toowoomba one training session was held for 30 participants.  It was found that the training for the larger group in Toowoomba was less successful than the smaller groups because it was difficult to maintain engagement with a group of that size.   The participants’ abilities to use the PDAs varied significantly.

The Project phase evaluation was based on:
•	Use and acceptance of the PDA by the students and academic staff.
•	The level of use (and hence relevance) of the software available to students and staff.
•	Assessment of the impact that such technology has on studying medicine in the rural environment.

A Project phase survey was undertaken by participants three months after the PDAs were distributed.  It was originally envisaged that monitoring software would be used to measure usage levels.   Despite working with a company who claimed to be able to provide monitoring software, it was inadequate.

The Project phase post-survey elicited a response rate of 68%.  Most respondents, 71%, were aged between 20-29, with 44% being males and 50% being females.

The majority of participants, 64%, used their PDA several times a day. Other respondents used their PDA at least once a week or less (14%), several times a week (10%) or once a day (12%).

When asked about the ease of using a PDA 46% of respondents indicated they learned to use it with some assistance and have required minimal support.  Thirty-two percent learned to use it with some assistance and did not require support.  Sixteen percent learned to use it without assistance and did not require support.  Only a small percentage, 6%, learned to use their PDA with some assistance and required support on a regular basis.

Fifty-five percent of respondents reported they experienced no problems using the technology. Of those who experienced difficulties using their PDA some of the most common issues included:
•	Synchronization – most of these were solved with assistance. Ongoing issue for a Mac user requiring additional software to be purchased.
•	Difficulties downloading related to the QH firewall issues.
•	PDA freezing.
•	Lost programs and data because the battery was allowed to go flat.

In relation to convenience of PDA use 82% of participants found their PDA very convenient (48%) or convenient (34%) to use.  Fourteen percent were undecided and 4% found the PDA inconvenient to use.  This response indicates the majority of participants felt their PDA was convenient to use in a range of environments to support their clinical experience and studies.

The majority of respondents indicated they were either comfortable or very comfortable using the PDA in front of patients, consultants, registrars/interns, medical students, nurses and other staff.  Detailed results are available in Table 3.

Table 3. Comfort level using the PDA in front of the various groups
Situation	Very uncomfortable	Uncomfortable	Undecided	Comfortable	Very comfortable	
Patients	8%	19%	6%	46%	21%	
Consultants	4%	4%	12%	47%	33%	
Registrars/interns	4%	0%	0%	48%	48%	
Medical students	6%	0%	0%	33%	62%	
Nurses/staff	4%	4%	8%	48%	36%	

Responses indicated that information resources are consulted most often to get factual data about a drug (88%), to get factual information about a disease or symptom(s) (82%), to increase knowledge on a subject (70%), straight after patient rounds (66%), during patient rounds (56%), as a study aid (56%), as a means to help with writing up patient notes (38%) and other uses (2%).

PDA functions used several times a day by respondents included the calendar (56%), drug references (46%), medical references (43%) and note taking (19%).  Of respondents 41% indicated that did not use the medical calculator at all, nor did 41% use their PDAs to record patient information.  In accordance with the pilot survey findings many respondents indicated the medical calculator was not useful. The recording of identifying patient details on the PDAs was not permitted.  Patient information is recorded on an official QH records management system.  The address/phone book function was used by 38% of respondents and not at all by 31% of respondents.  (Details Table 4.)  

Table 4. PDA Functions used
	Not at all	Once a week or less	Several times a week	Once a day 	Several times a day 	
Address/phone book	31%	38%	17%	4%	10%	
Calendar	10%	10%	16%	8%	56%	
Drug reference	4%	17%	15%	19%	46%	
Medical Calculator	41%	33%	13%	10%	3%	
Medical reference
(eg textbooks)	6%	9%	28%	15%	43%	
Recording patient information	41%	29%	8%	12%	10%	
Taking notes	25%	25%	17%	15%	19%	

The Oxford Concise Medical Dictionary was rated by 46% of respondents as being useful.  The Archimedes medical calculator received a varied response.  MIMS on PDA was found by most respondents to be very useful (56%) or useful (22%).   The response to MIMS Interact on PDA was less positive with 23% of respondents indicating they were undecided; though 48% did indicate that MIMS Interact on PDA was either useful or very useful.  Sixty percent of respondents indicated the Oxford Handbook of Clinical Medicine and the Oxford Handbook of Clinical Specialities were very useful resources on their PDA.  (Details Table 5.)

Table 5. Information resources
	Not useful	Somewhat useful	Undecided	Useful	Very useful	
Oxford Concise Medical Dictionary	8%	14%	10%	46%	22%	
Archimedes medical calculator	24%	22%	20%	22%	10%	
MIMS on PDA 	 6%	12%	4%	22%	56%	
MIMS Interact on PDA	17%	13%	23%	29%	19%	
Oxford Handbook of Clinical Medicine	6%	4%	 2%	28%	60%	
Oxford Handbook of Clinical Specialities	4%	 8%	2%	24%	61%	

Fifty-two percent of doctors and medical students agreed that the PDA improved their performance as a clinician and 63% agreed that the PDA optimised their access to contemporary information at the point of need.  Fifty percent of respondents strongly agreed that the PDA assists in organisational efficiency and 45% indicated that the use of a PDA improves access to information resources for students placed in remote or isolated areas.  The majority of respondents were either undecided (34%) or disagreed (30%) that the PDA enabled them to spend more time on the ward.  Forty-seven percent of respondents agreed that the PDA did help to address diagnostic questions effectively. (Details Table 6.) 

Table 6. Value of technology to doctors and medical students
	Strongly disagree	Disagree	Undecided	Agree	Strongly agree	
The PDA would improve my performance as a clinician?	0%	4%	20%	52%	22%	
The PDA optimises my access to contemporary information at my point of need.	0%	4%	4%	63%	29%	
The PDA assists in optimising my organisational efficiency.	2%	6%	14%	28%	50%	
The use of a PDA improves access to information resources for students placed in remote or isolated areas.	0%	6%	18%	45%	31%	
The use of a PDA enables me to spend more time on wards.	0%	30%	34%	26%	11%	
The PDA helps me to address diagnostic questions effectively.	0%	18%	20%	47%	14%	

The majority of respondents (51%) answered between 5-10 clinical questions a day and 22% answered between 3-5 such questions a day.  (Details Table 7.)

Table 7. Clinical questions answered each day
No. clinical questions	0	1	2	3-5	5-10	11-15	16-20	>20	
Percentage respondents	2%	0%	2%	22%	51%	10%	4%	8%	

When asked “Do you think that access to the reference materials on the PDA contributes to your educational experience, why or why not?”  86% (42/49) of responses where positive. The most common thread (50%) related to the ability to look up information at the point of need and in the correct context. For example:
•	“Yes. It can make clinical situations into teaching moments when questions can be explored with the clinician in combination with PDA resources”
•	“Access information quicker and still in the clinical context, therefore it is more relevant. I am more likely to look something up if I don’t have to remember to look it up later”
•	“Allows me to find the answers when I want to know them – I would have probably forgotten about them by the time I got time to look them up otherwise”
•	“Yes, because if you are able to look something up while you think of it you learn more”
•	“My PDA allows me to look up more of my questions as they arise rather than having to remember to look them up later when often it doesn’t get done”
•	“Allows me to get an answer to my clinical questions on the spot”
•	“Contributes by allowing me to look up information when the need arises, thus providing a more salient learning experience”
•	“Yes, definitely. It enables answers to some questions immediately – particularly for drug information or quick reminders of different diseases, so can better apply knowledge to that patient straight away”

Other recurring comments included:
•	“MIMs is especially useful, because pharmacology is not well covered in the course itself and I always have thousands of questions”
•	“Easily accessible for a topic that needs a quick refresher, and if I have a spare 10 minutes between items then I use my PDA for study on various topics”

Fourteen percent (7/49 responses) were negative. Reasons included:
•	“Handbooks in print form and they are much easier to use”
•	“At times it is INCONVENIENT (and/or rude) to try and use a PDA in a clinical setting”
•	“Don’t like the texts that have been included”

When asked on average, what percentage of clinical questions do they seek answers for at the point of service, 64% of respondents indicated they sought answers at the point of service for more than 50% of questions.   Fourteen percent indicated they sought answers at the point of service for less than 50% but more than 5% of questions.  Twenty-two percent indicated they were seeking answers at the point of service for less than 5% of questions.  

When asked on average, what percentage of clinical questions do they seek answers for at a later time, 72% of respondents sought answers to less than 50% of questions at a later time, 28% of respondents sought answers to more than 50% of the questions at a later time.  Sixteen percent of respondents either did not seek answers at all or did not understand the question.  Not seeking the answer may mean that they were confident they knew the answer. The response to the question about the number of clinical queries to which respondents sought answers suggests three types of users: one group who used the PDA almost all the time, a second group who used it 50% of the time and a third group that did not use their PDA at the point of care.

Respondents were asked to rate the importance of their preferred resources used to answer clinical questions, from 1 to 5, with 1 being the most important to 5 being the least important.  The following is a summary of the most significant results:
•	54% rated attending physicians as 1 rating
•	36% rated print or web based medical texts/references gave a 1 rating and 26% gave a 2 rating
•	34% rated their PDA/Handheld computer as 2
•	28% rated residents/interns as 2 and 26% rated residents/interns as 1
•	28% rated other students as 2
•	26% rated electronic databases as 3

The following is a summary of written responses to the question: “List any reference materials, administrative functions (e.g. academic calendar) and teaching and learning resources (e.g. logbooks and PowerPoint presentations) that you think should be added to the PDA.”  A summary of written responses revealed the following recurring (greater than 1) suggestions:
Table 8. Other resources required
Title	No. of responses	
Australian Medicines Handbook 	9	
Harrison’s on Hand	9	
eTG (Therapeutic Guidelines)	8	
UpToDate	5	
Murtagh, J  General Practice (not yet available on PDA)	4	
General anatomy resource	3	
Academic calendar (Rockhampton only)	3	
Clinical Logbook	2	

Participants were finally asked to comment on the positive and negative aspects of the PDA Project. The three main positive themes that emerged were:

1.	Access to information at the point of need (feedback similar to that already reported)
2.	Organisational benefits. Comments included: “having phone book, calendar, note pad, and reference books all in one device is fantastic, I don’t need to take anything else with me to the wards” and “the usefulness of the PDA for not only medical things, but also organisation”
3.	Opportunity to trial the technology. Comments included: “Exposure to the use of such devices as I believe medicine will eventually be run by such technology on the wards. I’ve noticed that many of the residents / registrars have or are in the process of buying PDA’s or palm pilots for work” and “Encouraging use of IT in clinical practice at a time when, as a student you are still working out a routine and are able to incorporate the use into your practice”

Negative comments were fairly benign, with the main themes being: having to give the PDA back; fear of flat battery and data loss; and PDA envy that is Consultants and Registrars being jealous of a student with a PDA.

Discussion
The Project’s objectives were addressed in the following ways:
•	Within budget constraints, a limited set of evidence based information resources were provided to the RCD medical students at their point of need.   The survey has identified that some of the resources were useful for answering clinical questions (e.g. the Oxford Handbooks and MIMS).  It was identified that students would like additional PDA resources, such as the Australian Medicines Handbook, Harrison’s on Hand, therapeutic guidelines and UpToDate.  Provision of resources for PDA use in a clinical setting requires periodic reassessment; especially as new PDA resources are constantly being released on the market.
•	The survey results indicate that the PDAs have contributed to creating a positive rural medical education experience, with 45% agreeing and 31% strongly agreeing, the PDA improved access to information resources whilst working in a remote or isolated area.  It has been identified that PDAs are helping to attract students to RCD  placements, as the students are keen to gain experience using this technology.  Further exploration of teaching and learning applications of PDAs within the School of Medicine and in the tertiary education sector will continue. The School of Medicine plans to develop clinical electronic-logbooks for the collection of clinical competencies using the Internet and PDAs as the collection media. 
•	The PDA Project has successfully helped to optimise contemporary information and telecommunications in healthcare and prepare students for practicing medicine in the 21st Century.  It has demonstrated how the innovative use of technology can help to manage information and utilize the latest information at the point of need.  Ninety-two percent of the Project survey respondents agreed that this was the case.
•	The Project did provide a platform to test the applicability of PDA technology to the teaching and learning environment.  The strong enthusiasm of students to participate in the Project and the fact that 74% believed that such technology would help their performance as a clinician is testimony to the importance of introducing technology to enhance the teaching and learning process.
•	The UQ Cybrary did gain valuable experience by participating in the Project.  The experience identified the importance of the Librarian’s role to the assessment and negotiation for information resources, provision of training, technical support and management of the units.   

The success of the Project and the fact that PDAs are becoming more pervasive at hospitals supported by UQ Branch Libraries also raises a number of issues for the UQ Library. It is anticipated that the Library will be able to support PDAs in a number of areas including:	
•	Additional Evidence Based Healthcare PDA information resources.
•	The creation of a PDA compliant website (including catalogue).
•	The creation of PDA specific resources such as Evidence-based Healthcare guides.
•	The ongoing implementation of wireless infrastructure within branch libraries.
•	Lobbying and negotiation with information resource suppliers and publishers to address the lack of site licensing options for resources.
•	Lobbying to have one payment for information resources rather than paying a site wide subscription and a further PDA download charge. 
•	Provision of PDA training and education opportunities.
•	Provision of technical support and advice to users.
•	The development of a PDA Website to provide information and support to users.
•	The promotion of PDAs use through user groups.

Focus groups were being held at the time of writing to explore the themes emerging from the survey findings and to elicit other feedback.  Results from the focus groups and data from manual usage logs undertaken by participants will be reported in a later paper.  Once all the data has been analyzed at the conclusion of the Project in December 2004, the final report will include recommendations regarding the future use of PDAs at the School of Medicine and the UQ Cybrary.

Conclusion
The pilot phase was an extremely beneficial part of the UQ PDA Project, as it helped to fine tune technical and administrative processes vital to the successful outcome of the Project phase. Integral to the success and efficient operation of the Project was the appointment of a library staff member as the part-time Project Coordinator.  This person was dedicated to the role which was a significant advantage as the Project Coordinator was not distracted by other responsibilities.  The input from a range of staff with wide ranging expertise also contributed to the successful outcomes of the Project.   The relationship between the Library and the School of Medicine was strengthened as a result of the partnership.  

The positive results and feedback obtained in the final surveys are testimony to the value of adopting PDAs in a rural clinical teaching setting.  The trend identified was that most participants use their PDAs several times a day and can use the units effectively with minimal instruction.  Most participants found the PDAs very convenient and comfortable to use in front of a range of groups.  It was demonstrated that the PDAs were used for accessing a wide range of resources, including academic timetables, contact details, clinical facts and drug data.  UQ health librarians plan to continue their proactive role in helping to create an evidence-based training environment for all medical students, to ensure that good habits are formed early and continue leading to life long successful clinical practice. 

Keywords:

Personal Digital Assistants (PDAs)
Education, Medical, Graduate
Computers, Handheld
Program Evaluation
Rural Health

Acknowlegements: 

Janine Schmidt, University Librarian, University of Queensland Library

PDA Project Reference Group:  University of Queensland – Associate Professor Peter Baker, Dr Jeff Thomsett, Mr Andrew Heath, Ms Yasmin Childs, Ms Kerry Kruger, Ms Sarah Thorning, Ms Janelle Coe, Ms Donna Georgeson, Mr Paul Harmor (student representative) Dr Melita Cullen, Ms Lisa Kruesi and Queensland Health – Mr Graham Moore, Dr Brian Morris, Dr Andrew Cumming