God Machine has some wonderful suggestions. One that I don't quite agree with, though, is having the player alone as often as possible; that works as the game progresses, but in the beginning, I wouldn't have them alone. One thing that scares people is the sense of isolation; having them start with some decent interaction with others that slowly dwindles over time until they're left completely alone can be quite effective. You could have other characters assist your protagonist in some memorable way, then have the protagonist face a similar situation later in the game with no assistance whatsoever... makes it feel far more hopeless.
I also agree that having a strong character who is nonetheless overwhelmed is a fantastic idea.
The one thing that I haven't really seen mentioned yet, though, is pacing.
To me, games are far more terrifying if the tension builds slooowly.
For an awesome example of this, play the opening of Silent Hill 2 as James is walking through to the town. Absolutely stunning handling of atmospheric terror when there's absolutely nothing
that can actually harm you in any way.
I should also add that there are situations that are, in and of themselves, unsettling or disturbing. For instance, in my game, the player is ostracized from the town he lives in and forced into the woods where he meets a man who lives with the corpse of a young girl as his wife. The player will have to interact with this character often, entering and leaving his cabin, doing things for him because there's no one else around who can help him in any way. But what you have to do can be unsettling at times. This is also a good example of dwindling social interaction: my game starts in a populated town, moves to the outskirts of town, eventually into the woods, and keeps getting more and more isolated from there. Helps with that feeling of being alone to remember that you haven't been.
And often, this kind of atmosphere provides enough discomfort in the player to really get under their skin. There are also psychological games that you can play with the main character (for some inspiration, look up solipsism and dissociative fugues).
I'd also add that throwing in limited resources is extremely useful. If your game uses a flashlight, make batteries scarce. If there are weapons, make sure that the player will have to run more often than fight, so on and so forth.
Hope that was helpful.