Yeah, that. And I do find it striking that, for all Lovecraft's blatant racism and xenophobia, you do see an occasional moment of empathy, the most memorable being at the end of "Mountains of Madness" - my book is still in storage somewhere back in New York state, so I'm paraphrasing, but I believe the quote went something like "...monsters, star spawn, they were men! What were they but the men of another day and age, and what did they do that we would not have done in their place?"
I also find it fascinating to think about how "Shadow out of Time" would have gone if the body-swapping program had been voluntary rather than coerced (and can you believe I totally would have been jumping out of my seat to volunteer for it?), and part of a larger cultural exchange program between such different cultures and time periods... I really, really, really want to see this story written.