How to filter duplicate content?

[eluser]codex[/eluser]
For my affiliate webshop I have written a script that reads in a CSV file (datafeed) and inserts the contents into the database. Unfortunately the merchant that provides the datafeed has different categories in which the same product ID can exist, which means that product ID 1243 can exist x times because it's defined in (for example) category books, category dvd's and category cd's. I would like to filter the duplicate product ID's, but how do you do that?

[eluser]GSV Sleeper Service[/eluser]
if the product id is a PK, then you can use the IGNORE keyword when running 'load data infile', assuming you're running mysql.

[eluser]codex[/eluser]
[quote author="GSV Sleeper Service" date="1234304183"]if the product id is a PK, then you can use the IGNORE keyword when running 'load data infile', assuming you're running mysql.[/quote]

I'm not using 'load data infile' but 'fgetcsv'. I'm looking at 'array_unique', but I'm not sure how to implement that on the result array.
Another option is jsut to do the insert with duplicates and remove those after. But how do you say 'get all duplicate id's, then remove the duplicates but leave 1 unique'?

[eluser]TheFuzzy0ne[/eluser]
A good reliable way of doing what you want, is to simply order entries by ID, then iterate through each entry, comparing the ID of the current row to the last. If a matching row is found, it's down to you what you want to do with that row, and how you want to handle it.

Hope this helps.

[eluser]codex[/eluser]
[quote author="TheFuzzy0ne" date="1234315602"]A good reliable way of doing what you want, is to simply order entries by ID, then iterate through each entry, comparing the ID of the current row to the last. If a matching row is found, it's down to you what you want to do with that row, and how you want to handle it.

Hope this helps.[/quote]

Thanks, that was an option too indeed. But for what I want to do it 's as easy as adding a group_by to the query. Sometimes the soultion is just in front of you but you just can't see it :-)

[eluser]TheFuzzy0ne[/eluser]
Whoops. My apologies. I misread what you wrote.