@Will - really thoughtful question. It sounds like the first thing you may need to do is set up a better analytics solution. Are you able to figure out the total books checked out in a certain time period, number of users on your system, and number of users who have books checked out? If you’re currently not able to track basic user workflows and these metrics, then it’ll be really difficult to do any A/B testing, since you won’t have a baseline or a way to measure success.
Once you do have a better data collection solution, then you can start looking at the problems you’re looking to solve. Are there students that do tons of searches but never check anything out? You could test different tweaks to the search results pages to see what helps people find what they’re looking for. Or, do lots of students log on but never visit helpful parts of your system outside of search? You can A/B test ways to better highlight the different useful tools you offer.
Note that most basic solutions, like Google Analytics, will tell you what brought your users to your site (search, referring link, search terms, etc.). You can also set cookies or another method of tracking your users between search and checkout. The key here is that you need more data - I hope that helps!