I love short stories, as well as novellas and novelettes, I suppose - English is not my first language and I'm not sure where to draw the line between them. So I want to contribute to this thread, and I do have some things to say, with the caveat that I have read the things I will talk about a good while ago and have not revisited them recently, so you are effectively hearing the impressions of a younger me, who in a way used to be a different person (certainly less aware about things that I now would look at critically etc).
Some of my favourites
Some of the first I paid lots of attention to were Asimov's, who you've already mentioned. Growing up I was a huge reading nerd and I came across a collection of his short stories when I was around 12 or so - I remember being so surprised that he can set up something intriguing in barely 3 pages, seeing that I was used to longer books. The favourites I have as an adult are probably George R R Martin and William Gibson, who both write sci-fi, but different flavours of it.
GRRM's are mostly sci-fi horror (or at least lean in a dark direction) but occasionally have striking insights into human psychology beyond what you normally expect from the genre. "Song for Lya" is way up there in my top favourite writings ever, but many of the ones grouped in "Nightflyers" and similar volumes, as well as his Tuf series, are worthwhile imo (though warning for lots of gore and some unsettling adult themes, less so than in his ASOIAF books though - I far prefer his short form writing, and his sci-fi to his fantasy, for several reasons).
Gibson's short stories are varied, some are fast paced and action-y, other are downright contemplative and romantic. Something about them is very immersive. In most of them, I find a very understated bittersweet sadness related to living in a world that's exciting and stimulating, yet occasionally (or maybe often) still leaves you without the things you need or desire most. I find that very relatable.
Why do I think short stories work
I suppose I can't speak for others, but in my case, the fast paced writing makes me feel energised. Also, the frequent lack of descriptions when it comes to either large parts of the setting or the characters allows the reader's brain to fill in the blanks with what the reader finds interesting. For for sci-fi, this often means that the shorter the story is, the better it ages, because it avoids avoid describing too many details that become dated faster than the author imagined (whether it's technology or social norms).
A question
Many years ago I read a short story online. The main character is a woman, and we see her adapting to technology making leap after leap (including some that allow her, as well as other people, to have a longer-than-currently-possible lifespan, as well as interstellar travel), and who then becomes the first ambassador for an encouter with a species of intelligent extraterrestrial beings. I found it incredibly heartwarming because as you first start reading it, you fear it will be a sad story about someone dying just as humanity is about to take a big technological leap, but it doesn't go in that direction. Instead, technology helps her overcome her health problems, she becomes closer to her family who lives far away, and becomes more and more adventurous and curious about the world in her very old (yet more active than ever) age. It was very uplifting and probably the only explicitly utopic (no darker hidden side) story I've ever come across.
I can't remember its name or find it again. Can anyone help?