Making Library Assessment Work
Practical Approaches for Developing and Sustaining
Effective Assessment
Steve Hiller
ARL Visiting Program Officer
University of Washington
Jim Self
ARL Visiting Program Officer
University of Virginia
University of Illinois Urbana-Champaign Library
May 23, 2005
Rise of User-Centered Library Concept
and the Culture of Assessment in the 1990's
User-Centered Library
All services and activities are viewed through the eyes of the customers
Customers determine quality
Library services and resources add value to the customer
Culture of Assessment
Organizational environment in which decisions are based on facts, research and analysis,
Services are planned and delivered to maximize positive customer outcomes
User-centered libraries "collect data and use them as the basis for decision-making rather than rely on subjective impressions and opinions"
(Stoffle, Renaud and Veldof, "Choosing Our Futures", C&RL, 1996)
ARL New Measures 1999-
Key Areas
User Satisfaction/Service Quality
LibQUAL+
Measuring Library Service Quality Lyceum
Market Penetration
Ease and Breadth of Access
Library Impact on Teaching and Learning
SAILS, Learning Outcomes
Library Impact on Research
Cost Effectiveness
Library Facilities/Space
E-Metrics
MINES
ARL supplementary statistics
Why Assess?
Accountability and justification
Measure progress/achievement
Improvement of services
Comparison with others
Identification of changing use and needs patterns
Identification of questionable services
Marketing and promotion
Decisions based on data, not assumptions
Assumicide!
Common Assessment Barriers
Organizational culture
Lack of administrative support/leadership
Time/Money/Staff/Resources
Establishing priorities
Knowing what and how to measure
Inexperience, lack of skills and expertise
Understanding and presenting results
Sustainability
Using results to make changes and improve libraries
S. Hiller and J. Self, "From Measurement to Management: Using Data Wisely for Planning and Decision-Making" Library Trends, Summer 2004.
Making Library Assessment Work:
Practical Approaches for Developing and Sustaining
Effective Assessment
ARL program to assist libraries with moving assessment efforts forward
Recognition that libraries are aware of the value of assessment but find it difficult to sustain and use results
Conduct site visits to evaluate assessment and consult on developing practical approaches to effective assessment
Identify common barriers and facilitators to assessment
7 libraries participating in Phase I Spring 2005; up to 10 during Phase II (2005-06 academic year)
What Are We Measuring?
"Institutional assessment efforts should not be concerned about valuing what can be measured, but instead about measuring what is valued."
A.W. Astin, Assessment for Excellence, 1991
"What is easy to measure is not necessarily what is desirable to measure."
M. Kyrillidou, "An overview of performance measures in higher education and libraries", 1998
Prioritize Assessment Activities
Importance
Alignment with strategic goals, objectives and actions
Library and institution
Timeliness
Feasibility
Degree of user involvement
Expertise needed and availability
Cost and resources available (including staff time)
Ability to improve and add value to library services
"…but to suppose that the facts, once established in all their fullness, will 'speak for themselves' is an illusion."
Carl Becker
Annual Address of the President of the
American Historical Association, 1931
What Does it Mean?
Understanding Your Data
Scan results for basic overview
Frequencies, means, patterns, variation
Use statistical analysis that make sense
Qualitative information and comparisons provide context and understanding
Seek internal or external validation
Within same data sets or others
Identify what is important and why
Communicating and Using Results
Identify key findings, not all results
Mix text, data, and graphics
avoid jargon
add context
Know your audiences. Make it understandable
Prioritize potential action items and follow-up
Identify "Handoffs" to those responsible for action
Look for some easy "wins"
Quick, inexpensive, and noticeable
Report results
Effective Assessment
Easier Said Than Done
Libraries in many cases are collecting data without really having the will, organizational capacity, or interest to interpret and use the data effectively in library planning.
The profession could benefit from case studies of those libraries that have conducted research efficiently and applied the results effectively.
(Denise Troll Covey, Usage and Usability Assessment: Practices and Concerns, 2002)
Two Approaches to Effective and Sustainable Assessment
University of Virginia
Performance and financial standards
Compilation of data from varied sources
Centralized Management Information Services unit
University of Washington
User needs assessment
Large-scale cyclical surveys and ongoing qualitative input
Assessment distributed throughout organization
University of Virginia Library
Organizing for Assessment
Management Information Services unit
Established in 1996
Currently 3 staff
Resource for library management and staff
Advocates for sustainable assessment
Centralized data collection, analysis and compilation
Multifaceted approaches
Collecting the Data at U.Va.
Customer Surveys
Staff Surveys
Mining Existing Records
Comparisons with peers
Qualitative techniques
Customer Surveys
Faculty
1993, 1996, 2000, 2004
Separate analysis for each academic unit
Students
1994, 1998, 2001, 2005
Separate analyses for grads and undergrads
Faculty Priorities
1993 to 2004
Top Priorities at UVa
Comparison of Faculty and Students
Staff Surveys
Internal Customer Service
2002, 2003, 2004
1 to 5 satisfaction scale
Worklife Survey
2004
Agree or disagree with positive statements
Data Mining
Acquisitions
Circulation
Finance
University Records
Acquisitions Expenditures by Format
University of Virginia Library
Expenditures of UVA Academic Division
19892003
Other Academic Support (+200%)
Research (+219%)
Total Academic Division (+140%)
Libraries (+81%)
Instruction (+80%)
The Balanced Scorecard
Managing and Assessing Data
The Balanced Scorecard is a layered and categorized instrument that
Identifies the important statistics
Ensures a proper balance
Organizes multiple statistics into an intelligible framework
Metrics
Specific targets indicating full success, partial success, and failure
At the end of the year we know if we have met our target for each metric
The metric may be a complex measure encompassing several elements
Rationale for the BSC:
Getting Control of the Data
Focus
Balance
Assessment
Intelligibility
What Do We Measure?
Customer survey ratings
Staff survey ratings
Timeliness and cost of service
Usability testing of web resources
Success in fundraising
Comparisons with peers
Metric U.1.A:
Overall Rating in Student and Faculty Surveys.
Target1: A score of at least 4.00 (out of 5.00) from each of the major constituencies: undergraduate students, graduate students, humanities faculty, social science faculty, science faculty.
Target2: A score of at least 3.90 from each of the major constituencies.
Metric U.4.B:
Turnaround Time for User Requests
Target1: 75% of user requests for new books should be filled within 7 days.
Target2: 50% of user requests for new books should be filled within 7 days.
Metric U.3.A:
Circulation of New Monographs
Target1: 60% of newly cataloged cataloged monographs should circulate within two years.
Target2: 50% of new monographs should circulate within two years.
Metric I.3.B. Staff Survey Rating
of Internal Service Activities
Target1: A composite rating of at least 4.00 in the internal services survey, with no unit rated below 3.50.
Target2: A composite rating of at least 3.50, with no unit rated below 3.00.
Using Data for Results at UVa
Additional resources for the science libraries (1994+)
Major renovation (2001)
Revision of library instruction for first year students (1995)
Redefinition of collection development (1996)
Initiative to improve shelving (1999)
Undergraduate library open 24 hours (2000)
Additional resources for the Fine Arts Library (2000)
Support for transition from print to e-journals (2004)
Growing Assessment at UW
From Project-Based to Ongoing and Sustainable
Libraries' first strategic plan in 1991 called for survey as part of user-centered services philosophy
Initial large scale library survey done in 1992 as "one-time project"
Library Services Committee formed in 1993
Conducted in-library use surveys in 1993,1996, triennial survey in 1995
Developed Libraries Service Policy in 1995
Library Assessment Group appointed in 1997
Focus groups, observation studies, in-library and triennial surveys
Assessment efforts in other library areas 1997-
E-Metrics, usability, Web design
Library Assessment Coordinator (50%) appointed 1999
UW Libraries Assessment
Organization
Library Assessment Coordinator (50%)
Chairs Library Assessment Group (9 members)
Coordinates and develops broad-based user needs assessment efforts (surveys, focus groups, observation)
Encourages and supports other assessment work
Shared and Distributed Assessment Activities
*Usability (Web Services, Public Services)
*E-Metrics (Assessment, Collection Management Services)
*Management information (Assessment, Budget, CMS)
*Instruction (Information Literacy, Assessment)
*Represented on Library Assessment Group
UW Libraries Assessment Methods
Large scale user surveys every 3 years ("triennial survey"): 1992, 1995, 1998, 2001, 2004
In-library use surveys every 3 years beginning 1993
LibQUAL+ in 2000, 2001, 2002, 2003
Focus groups on varied topics (annually since 1998)
Observation (guided and non-obtrusive)
Usability
E-Metrics
Guided Observation
Bibliographic Database Searching
Faculty and graduate students search very differently than we think they should
Common observations included:
Prefer to use single keyword search box
Little use of Boolean commands
Limits or format changes rarely employed
Commands need to be on first page or lost
Visible links to full-text critical
Important features for librarians are not necessarily important to faculty and students
Focus Groups (2003)
Library Research Competencies of Undergrads
2 groups of faculty teaching upper division courses
3 groups of students taking upper division courses
Major Findings
Undergrads expect faculty to teach them library research skills
Faculty expect undergrad students to have those skills already
Undergrads have difficulty determining the library sources to use
Faculty "dumbing down" library research assignments and amount of course reading
The personal connection with a librarian remains important
UW Triennial Library Use Survey
Number of Respondents and Response Rate 1992-2004
Large number of respondents allows for analysis within groups
Physical Visits Down, Remote Use Up
UW Libraries Use Patterns 1998-2004
(% of each group who use library at least weekly)
Faculty Library Use Patterns 2001/2004
by Academic Area (Those who use libraries at least weekly)
Use Libraries at Least Twice Per Week
(Remote and In-Person) 1998 - 2004 by Group
Resources Used Remotely Twice Per Week or More: Faculty and Graduate Students
Print/Online Priority by Academic Area
Faculty 1998, 2001, 2004 (% in each group identifying as priority)
Overall Satisfaction by Group
1995-2004
Assessing Branch Library Viability
Data-Based Decision Making
Performance Measures developed in 4 broad categories
Use
Print items, photocopies, reference questions, gate counts
Primary user population
Faculty, grad students, undergraduates, change over time
Facility quality
Quality of space for users, collections, and staff
Physical library dependency of primary user group
Importance of books and older journals, use of other libraries, frequency of in-person library visits, core journals available electronically
In-Library/Reserve Use 1997-98 to 2003-04
Science (5) and Fine Arts (4) Branch Libraries
Science/Engineering Faculty
Libraries Used Regularly (2004 Survey)
What We've Learned About/From
the UW Community
Libraries remain very important to learning and research
Library needs/use patterns vary by and within academic areas and groups (e.g. faculty and undergrads)
Library as place remains important to undergraduates
Faculty and students use libraries differently than librarians think they do (or prefer them too)
Library/information environment is perceived as too complex; users find simpler ways (Google) to get info
Remote access is preferred and has changed the way faculty and students work and use libraries
How UW Has Used Assessment
Information
Make our physical libraries "student" places
Identify student information technology needs
Move rapidly to desktop delivery of resources
Enhance resource discovery tools
Provide standardized service training for all staff
Stop doing activities that do not add value to users
Consolidate and merge branch libraries
Improve internal and external communications
Moving Forward
Keep expectations reasonable and achievable
Use data wisely and appropriately
Don't do more than you can support or utilize
Don't expect perfection; strive for accuracy and honesty
Assess what is important to the library, the institution, stakeholders, AND the customer
Use the data to make your library better
Keep your staff, customers, and stakeholders involved and informed
Retain Focus On The Customer
For more information…
Steve Hiller
Jim Self
ARL Assessment Project