I agree with JoB. In the sagas, the loss of Fylgja is usually a far more serious affair, distressing to both parties. It happens in rare instances where the behaviour of the human has so disgusted their fylga that she abandons them. Those people, when they die, can become the Icelandic style of evil ghost, quite material, capable of killing humans and other things, and very willing to do so. The kind that turns up in the Sagas, and in the rune list there is one to be used 'When troublesome ghosts/ ride the rafters aloft/ I can make it so that they wander astray'.
But usually the Luck doesn't leave a human until death.