I have added some child categories to a main category.
When I select the main category in the backend to show all the child category instance listings no instance gets shown.

Hello. Please give me more information. It is not clear what you mean.

For instance:

Main category in OS CCK is "car"
1st child category is "sports car"
2nd category is "family car"
3rd category is "land rover"
4th categrory is "city car"

In the backend in the menu "Instances" I have many instances of cars which belongs all to one of the child categories.
If I filter the instances now with the parent category "car" no instance get shown.
But I guess, if I click the filter for the parent category, all instances of the child categories should be shown.

Is it right?

Hello. No, you're wrong. This option exists only for the frontend. When you create a category layout, you can specify whether to display instances from the Child Categories. For the instances manager in the admin side, this does not work. Only instances from the selected category are displayed there.