There's a few theories. It's either related to the word for "bay" Vik , as they sailed out from bays, the old name for the coast near Oslo Viken , or according to Else Roehdahl it's a west-nordic word that meant something along the lines of pirate or sea warrior.

In the sagas it was mainly used about the guys going to war across the sea.