Your body regulates it. From what I've heard, saturated fat actually affects cholesterol levels more than dietary cholesterol itself. You should be fine. The only case where this is not true is familial hypercholesterolemia.
I actually found a paleohacks post yesterday where a guy LOWERED his LDL by eating MORE cholesterol:
LDL high? Increase your cholesterol intake. The elephant in the room? - PaleoHacks.com
Anyway, unless you have some really funky numbers or the total gets above 275 or so, I would not worry at all.