Hello
For show override layouts from
Joomla Book Library extension you need create layout in "templates/{your template}/html/com_booklibrary/" and copy there views from folder {your site}/components/com_booklibrary/views.
But please check if you need create override for layout "category view".
You need file:
{your site}/components/com_booklibrary/views/showCategory/tmpl/default.php
copy to
{your site}/templates/{your template}/html/com_booklibrary//showCategory/default.php
or you may
{your site}/components/com_booklibrary/views/showCategory/tmpl/default.php
copy to
{your site}/components/com_booklibrary/views/showCategory/tmpl/default_new.php
and select new layout in settings
After that, Please clear in Joomla cache,
Please in joomla admin menu: System->Clear Cache
Please in joomla admin menu: System->Clear Expired Cache
After please press Ctrl + F5
Regards
OrdaSoft team