I added more books and everything seemed to slow way down again, here is the response from my host so you may be able to adjust the code or table setup in the future:
That same query (or one very similar to it) is causing big mysql load. I have created indexes on the affected tables, that should speed the query. I have also restarted mysql to kill off any old, hung queries. For your reference the problematic query seems to be:
SELECT blc.id, blc.name, bc.catid, blc.parent_id FROM jo25_booklibrary_main_categories AS blc LEFT JOIN jo25_booklibrary_categories AS bc ON bc.catid=blc.id LEFT JOIN jo25_booklibrary AS b ON bc.bookid = b.bookid WHERE blc.section = 'com_booklibrary' AND blc.id = 55;