Yes, you are right the calorie charts tend to not be very accurate for people at either end of the bell curve in height. The same has been seen by some very petite women on this thread.
Otzi also makes a valid point that the official charts are also calculated with a sugar burning junk foodie in mind. My advice would be to track your calories (not restrict but just track) for a while. See what happens. You will find the right spot where you lose weight. All the charts say I should be eating at least 2000/day but I know my maintenance level is 1500. Trial and error and then trial and getting it right.