Armor consists of two parts: Armor type and armor value. The former, armor type, is fixed for each unit (a hero will always have armor type Hero) during the whole game. The used armor types in Dota are: Hero, Unarmoured, Fortified, Light, Medium and Heavy. It is the combination between Armor type and Attack type which determines the primary reduction in damage. Which types are reduced can be seen in the tooltip which appears when you hover the armor type icon ingame.
Armor value can be changed while playing by various items and abilities. Some increase the units' armor value (Plate Mail) and some reduce it (Acid Cloud). Only damage with Damage type Normal will get reduced by armor value. Negative armor has a cap of -20 (71% damage increase). Reducing the armor even more will not change the damage increase.
Despite the total damage reduction lowering the more armor you have, every armor point adds a constant 6% additional effective HP.