I've been looking for a solution to this all over the place and I cant seem to find a solid good solution.

What I want to do is take a large text file and remove the duplicates. (sorting is unimportant but I wouldn't mind learning how to do it both ways) Everytime I google for some sort of answer I keep getting other's crappy programs from sites I don't know and can't trust. So I went to a few of the c++ forums that I know and searched up and down for something similar to this problem but came up empty handed in every forum I came across. Am I overlooking something soo obvious that not even a programming noob has a problem with?

Just a push in the right direction would help immensely. Right now I'm confused as to whether I should be putting the input in strings or char array's with such a big list. vectors also come to mind here but that would be one huge vector.

I'm actually writing this is visual c++. I'm fairly new to c++ itself. Making the transition from vb to C++ right now. I'm fairly proficient in vb.