Paradoxically, while the French national anthem indeed has some rather enthusiastic bits about battle, it's not very nationalistic; there isn't much about France being great in it, rather it's more about defending our rights and freedoms against oppressors and tyrans.
While I don't know the lyrics beyond the first verse, I would not feel weird singing it, because it doesn't have that "our Nation is so great and others suck" vibe to me.