I believe part of the confusion stems from our "familiarity gap". We're familiar with the range of features within our own sphere. But when we meet someone from a race that is different than the one with which we grew up, we cannot distinguish (at least not at a moment's notice) the range of features within that group. Our minds have not made note of nor filed refereece to those differences and so it is difficult to identify them. Nothing offensive is intended. It is a failing in our brain that we wish were not so. But there it is.