Generally armor/magic pen will help against low armor/magic resist enemies more, than high armor/magic resist enemies.
+55 magic pen with a reduced 20 magic resist to surrounding enemies. (marks, quints, guise, abyssal scepter, and sorcerer shoes)
Having those stats and using a 100 damage spell to attack an enemy with 50 MR, they would take 120 damage instead of 67.
Having those stats and using a 100 damage spell to attack an enemy with 100 MR, they would take 80 damage instead of 50.
Having those stats and using a 100 damage spell to attack an enemy with 150 MR, they would take 57 damage instead of 40.
Having those stats and using a 100 damage spell to attack an enemy with 200 MR, they would take 44 damage instead of 33.
To make the wall of text less wall like
I am breaking this up a bit.
So, at 50 MR one would do 79% more damage.
At 100 MR, it would be 60% more damage.
At 150 MR, it would be at 42% more damage.
At 200 MR, it would be at 33% more damage.
Lets compare it to how much AP you can get with how much gold needed to build that.
About 286 AP instead of 95 AP with the pens, for the same amount of gold. (around 5.2k)
And if your skill has a .3 AP ratio it would be giving you...would add 57% more damage than your base spell damage. And since most enemies spells have a better ratio than .3, once they break 100 MR you should invest in AP. (though it might last a bit longer on the gold efficiency scale if I didn't include the abyssal scepter)
And now for spell pen %. You can get 50% with masteries + void staff.
Having those stats and using a 100 damage spell to attack an enemy with 50 MR, they would take 80 damage instead of 67.
Having those stats and using a 100 damage spell to attack an enemy with 100 MR, they would take 67 damage instead of 50.
Having those stats and using a 100 damage spell to attack an enemy with 150 MR, they would take 57 damage instead of 40.
Having those stats and using a 100 damage spell to attack an enemy with 200 MR, they would take 50 damage instead of 33.
Giving that % magic pen only starts to do better than flat magic pen when opponents are over 150MR.
Lets do the same AP thing here.
19% more damage at 50 MR.
34% more damage at 100 MR.
42% more damage at 150 MR.
51% more damage at 200 MR.
About 104 AP to 74 AP, with a massive 1.0 AP ratio that would add 30% more damage to your base spell damage. (the gold value was around 2.2k)
Looking at the numbers I posted, it appears that magic pen is the most efficient damage boost for the same amount of gold by 150MR. But giving the other utility you get off of the flat magic pen it is like comparing apples to oranges when considering gold costs.
This post might be useless, but I shall post it anyways. >: D