The Surprise Roll

Sometimes an encounter, either random or planned by the DM, catches one of the two groups involved totally off guard. This is called surprise and is determined by rolling 1d10 for each side (or only one side if the DM has decided that one of the sides cannot be surprised, for some reason). If the die roll is a 1, 2, or 3, that group or character is surprised.
Naturally, surprise does not happen all the time. There are many easy and intelligent ways it can be prevented. The most obvious is if the player characters can see those they are about to encounter well before getting close. For example the characters may see the dust of a group of horsemen coming their way, or notice the lanterns of a group of peasants coming through the woods, or hear the grunting barks of a gnoll war party closing through the trees. In these there is no way the characters are going to be surprised by the encounter. But if a leopard leaped upon one of the group while he was intently watching the approaching riders, or if a group of goblins suddenly sprang from the darkness, then the characters would have to roll to see if they were surprised. They were unprepared for these threats and so could be taken off guard.
The DM decides when a check for surprise must be made. He can require that one roll be made for the entire party, that a separate check be made for each character, or that only specific characters check. This depends entirely upon the situation.
For example, the entire party is intently watching the band of approaching riders. Then a leopard leaps from the branches of a tree overhead. The DM knows that no one in the group was particularly paying attention to the treetops, so he has one person in the group roll the Surprise die for the entire party. The roll is a 2, the PCs are surprised, the leopard gets a free round of attacks, and there is mass confusion as the clawing, biting creature lands in their midst! If two of the characters had been on a general watch, the DM could have had these characters roll for surprise instead of the entire group. If both were surprised. the entire group would have been unprepared for the leopard's attack. Otherwise, one or both of the guards might have noticed the creature before it pounced. Experienced player characters quickly learn the value of having someone on watch at all times.
The surprise roll can also be modified by Dexterity, race, class, cleverness, and situation. Modifiers can affect either your character's chance of being surprised or his chance of surprising others. A plus to your die roll reduces the odds that you are surprised; a minus increases those odds. Likewise, a minus to the enemy's die roll means that the modifier is in your favor, while a plus means that things are going his way. High Dexterity characters are virtually unsurpriseable, caught off guard only in unusual situations.
It is important to bear in mind that surprise and ambush are two different things. Surprise works as explained above. An ambush is prepared by one group to make an unexpected attack on another group and works only if the DM decides the other group cannot detect the ambush. A properly set ambush gives the attackers the opportunity to use spells and normal attacks before the other side reacts. If the ambush succeeds, the ambushing group gets its initial attack and the other group must roll for surprise in the next round, so the ambushing group may get two rounds of attacks before the other group can reply.

Effects of Surprise

Characters and monsters that are surprised all suffer the same penalty. They are caught off guard and thus cannot react quickly. The surprising group receives one round of attacks with melee, missile, or magical items. They cannot use these moments of surprise to cast spells. A ranger on the unsurprised side could fire his long bow twice (two attacks per round) before his opponents could even hope to react. A fighter able to attack twice per round could attempt both hits before any initiative dice are rolled. A mage could unleash a bolt from his wand of lightning before the enemy knew he was there. Of course, what applies to player characters also applies to monsters, so that the leopard in the earlier example could claw and bite before the characters even knew what was happening.
The second effect of surprise is that the surprised characters lose all AC bonuses for high Dexterity during that instant of surprise.
The surprised characters are dumbfounded by the attack. Instead of ducking and countering, they're just standing there rather flat-footed (maybe even with dumb expressions on their faces). Since they don't grasp the situation, they cannot avoid the hazards and dangers very well.
Surprise can also be used to avoid an encounter. Unsurprised characters can attempt to flee from a surprised group before the other group reacts. Of course, this is not always successful. since escape is greatly dependent upon the movement rates of the different creatures. If both groups manage to surprise each other, the effects of surprise are cancelled.
For example, Rath runs around the corner straight into some lounging guardsmen. Taken by surprise, he stops suddenly and frantically looks for someplace else to run. The guardsmen in turn look up rather stupidly, trying to figure out why this dwarf just raced around the corner. The surprise passes. Rath spots another alley and the guards decide that since he's running, Rath must be a criminal. Initiative rolls are now made to who acts first.