Or, overturned DM expectations and what to do about it.
This session I tried to run a scary encounter. I set the scene by having NPCs in town describe the place the PCs were going as deadly. The locale was a half sunken derelict ship and I happened to roll bad on the weather chart so they boarded the ship under dark clouds in rain. All was going well and at first it worked. The monster creeped out of the dark, the players made the quick decision to run, everyone was engaged. But then a regular combat broke out. The players got some good rolls and they made their saving throws against what I thought would be its devastating attack. The monster, injured and inept, slunk back into the darkness and the PCs when on with their looting. (you know, like they do)
Later I thought about the previous session when a random monster encounter at night (a throw away encounter really) turned into a tense and deadly fight ending with the Magic-User casting her most powerful spell. The PCs were aboard ship and at anchor at night, when sea scorpions crawled onto the deck. Those scorpions got a string of amazing rolls. There were five of them and they rolled 4 or 5 natural 20s in two rounds. 20s are criticals in our game for both the PCs and the monsters.
In light of the two above contrasting examples, I've got some advice for myself and anyone else who has had similar experiences.
How to make putz monsters entertaining but not necessarily deadly:
I've made inconsistent efforts to give monster "maneuvers" to spice up combat. Maybe for putz monster encounters a natural 20 is a good opportunity to exercise this. Since my Players are used to monsters getting the same critical damage they get, I don't think they'll complain too much when something besides them loosing extra HP happens. In the example above, the scorpions could have grabbed the characters and started to drag them off the deck. That would have made the fight more gripping (ahem) and less deadly at the same time.
Monsters are Dumb:
The really scary villains are the smart ones. If some monsters aren't supposed to be all that scary by all means make them dumb. For example, when the sea scorpions rolled the first 20 against the hireling, sure that drops him but then they all move in to eat him and/or fight over the food! That makes it easier for the PCs to take them out with out putting on kid gloves, because mechanically it may be less deadly (in terms of the number of attack rolls made against the PCs) but it's still grisly.
How to make the Big Bads truly Big and Bad:
Multiple lower powered monsters can be more deadly than a single Big Bad because of the sheer number of attack rolls. One way to address this would be to give the Big Bad an area attack. Outside of a dragon's breath weapon this is 4e style thinking that I'm going to try out in my B/X game. Sacrilege!
Dead if you don't Save, half dead if you do:
If a monster is really fearsome and it's attack causes a saving throw to be rolled, something bad happens even if you make your saving throw. The first PC who makes a successful save may be incredulous, but if you tell them something worse would happen if they hadn't made the saving throw, then they'll be scared. And there's precedent in old school D&D, half damage from a successful save vs. dragon breath.
How to make things scary:
Do it like the movies:
Scary things are half seen in the dark before they attack. Give them the chance to run, struggle with the door, all the horror movie stuff, and when they attack they're deadly fierce every single time. See above.
When all of the villagers tell the PCs how terrible and nasty the thing they're heading of to face is they'll listen. Even if they don't, they'll remember what the villagers (or whoever) said when things go south.
This is a DM's diary kind-of post. I realize none of this is a major revelation but after having both of these encounters go the exact opposite of the way that I intended or thought they would I wanted to collect my thoughts. Of course it's not the DM's job to be set on encounters going a certain way but I think if I can implement the above I'll be able to make encounters more consistently exciting and interesting without tweaking die rolls or sacrificing player agency.