I also have a large library of perhaps 120,000 records. I don't know if Joomla, or Book Library, can handle so many records efficiently. But Joomla can handle large forums so I suppose 120,000 book records is not very large compared with a busy forum.
crossmr, I have done a lot of work in the past, with importing and exporting data. My advice is to break down your spreadsheet (or import files) into smaller chunks. You will most certainly have problems with such a large import. The import could timeout, or get corrupted.
It only takes one little thing in one little record to ruin an import in process. It is much easier to narrow down problems within the data, when you import in smaller pieces.
Importing the records is one thing, it is a separate problem all by itself.
Maintaining the records online is another thing. From what I have learned, looking at Joomla forums and other forum comments about PHP, Apache & Linux: your library of 20,000 books should not be difficult for Joomla or Book Library to manage, IMO. But it also depends, in part, on the equipment that is hosting your site, whether there are limitations on the size of mysql table, how fast the server is etc. Speed and capability do not depend on the software alone.
I do think this is a very professional program: the programmers have done a very good job.