It is harder than it seems to make a tale believable and hats off to fantasy writers who manage to achieve this despite fanciful creatures such as dragons, demons, elves etc. But, strangely, authors in other genres also sometimes fail to make their world believable. I have given up on several historical fiction books, because they simply don't feel right. A recent novel about Rome was written in a modern style and perhaps that was what the author intended, but for me that failed utterly. An Egyptian novel I read focused on a few main characters and didn't mention anyone else. It felt lifeless whereas a Wilbur Smith tales, set in the same period, had a cast of thousands and felt more believable for it. This was achieved in a few sentences, adding to the clamour of life.
In all writing, I feel that you should be able to see, smell and even taste the world that you are in. The author needs to translate their knowledge of, say, ancient Rome to the reader and that means following their characters through crowded streets, filled with everyday folk and with ex-gladiator guards forcing passage for some high ranking noble, vendors screeching out their wares of Garum, pots and pans etc. and the smell of open sewers competing with that of roasted dormouse. The clothing, the food, the sights and again the smells are ways of convincing a reader that the tale is real.
In the most challenging genre, fantasy, how then do authors achieve believable tails? In my view it is characterisation that often achieves this. Also, no tale should have an easy and obvious solution, like a wizard destroying everything in his path. Characters need to be vulnerable in some way. Gandalf in Lord of the Rings is a great example. He is a key character and yet he is flawed in so many ways, e.g. blind to Saruman's deceits and facing a balrog, his best defence being a spell to block a door; a simple and yet effective use of magic. There is never a fiery blast that destroys thousands.
Fantasy and Sci-Fi tales must have bounds and rules. Star Trek was awesome and the rules there were not too far fetched and some even have become common place in modern society. For example, communication devices that fit in the palm or lapel. The more extreme deviation from modern life were energy shields, Warp engines and matter transporters. Their description and manner of employment, however, made them convincing. Each technology had believable faults that we could relate to e.g. a planet's atmosphere interfering with comms, energy shields that could withstand only so many hits and drained power, and matter transporters that simply failed (as in being spliced - Harry Potter books).
In fantasy the same issues of vulnerability help to make the tale believable. In Lord of the Rings Smaug has a displaced scale, making him vulnerable while his insufferable ego and greed add to his character.
Having a well thought through history of your world also makes fantasy and Sci-Fi believable in the same way as historical fiction needs a background setting. For example, readers need to know why a battle is about to happen, or the politics of races that set them apart. Why, for example, did some races embrace Rome and yet some defy her?
Making tales believable is key to being a good author. I do hope that in my tale, the Prophecy of the Kings, I achieve that despite such strange creatures and settings.