I'm not particularly into horror, it's not my thing, but I am into film and appreciate a movie that's well made and offers a bit more than your typical Oscar fare.
With that in mind I've been watching a variety of horror films, expanding my genre, as well as revisiting some that I've seen a few times before. And I've come up with a few notes as to what I think works (and what doesn't).
#1 - There should be some invocation of the Supernatural - demons, vampires, ghouls, beyond the grave sort of shenanigans. There should be more than the fear of death. That said - I'm going to confess, some of the scariest movies I've seen haven't involved that at all - I'm thinking here of "The Talented Mr. Ripley", which I found very disturbing and had no requirement or need of any supernatural elements. On that note - the reason I found Ripley such a terrifying antagonist is, I think, in the fact that he - the archetype, the amoral, charming sociopath - is so well portrayed and recognizable and - like it or not - we've all met a few, although circumstances might have allowed us to survive.
#2 - They come from the ordinary life. The best horror movies are set in the present - providing an easy reference point for the audience - films that are set in the past or future - eg: "The Village" or "Alien" require an unnecessary leap of the imagination. Set them in the present. "The Haunting of Hill House", "The Exorcist", "The Shining"; they're all great movies, and they were all set in the present. If horror can be presented or framed in the Ordinary, the Mundane, the repetition of dull routine and events it's infinitely more relevant and relateable. This applies as well to the possession of commonplace objects - dolls, etc. - in "Poltergeist" the possessed clown doll common in the 80's, the face of Chewbacca on the back of a jacket or lunch pack, a neighboring tree.
#3 - As little CGI or Special effects as possible. Real Horror depends on the suggestion of terror, the implied, the unknown. Making visible or obvious the threat (s) diminish them. Terror is unique, personal, and best left to each viewers individual imagination. Perhaps this is why "Poltergeist"- while a fun and suspenseful movie with lots of jump scares - doesn't really leave you with any great feeling of terror or fear. The same rule applies to Gore - as little as possible, hinting is the best form of exposition.
#4 - The realization and acknowledgement of a secret order of forces that are hostile to ones intent. Think "Wicker Man", or any horror film, in which the protagonist realizes he or she is up against forces she didn't expect or even realize existed. The gradual reveal of a hitherto unexpected or unanticipated plane of thought or being. A dark sort of enlightenment or realization that the universe is not entirely a friendly place...
#5 - The Use of NPC (Non-presented characters). Think Regan's "Captain Howdy" in "The Exorcist", Danny's friend Tony in "The Shining", the unseen hand held in "The Haunting of Hill House" - all foreshadow that things are about to go a bit sideways....
#6 - Like #3, the gradual building of suspense through (inexplicable) events that transpire off-screen or camera. The slow suggestion that the protagonist and we, the audience along with them, are going into uncharted territory.
#7 - Foreshadowing, this, as with any film; we see the knife innocently chopping food that will later be wielded to do us harm, we see the children playing with toys that will become later possessed by forces we don't understand, conversations that suggest what will be to come, hints, clues, for the alert audience members.
#8 - Not merely the fear of death, but that of Spiritual Annihilation. This is no mortal terror. In "The Shining" Wendy's fear is less for her and Danny's life than that she will be bound to the Hotel if she dies there. "Get Out" and "Don't Breathe" - while not invoking the supernatural, do invoke the "Binding to Place" - the threat that this situation could continue beyond the duration of the film, that the protagonist becomes powerless. Think of Vampires - what is there to fear about immortality? It's the fear that one endures, but not as oneself, not on one's own terms. The same with Werewolves. And in "The Exorcist" they are not just saving Regan's life, they are battling for her soul.
These are just loose notes, ideas, but Horror, as a genre, seems best described as a sort of Hero's journey in which the Hero gains admission to a higher order of knowledge or spirituality at a price that may not allow for his or her return, and if they do return their outlook on the world is forever changed...