Remove Duplicate Values From List Of Dictionaries

I'm trying to filter out my list of dictionaries by two keys. I have a huge list of items and I need to find a way to filter out those items that have repeated 'id' and 'updated_at

Solution 1:

Instead of a list of dictionary, why not a dictionary of dictionaries?

filtered_dict = {(d['id'], d['updated_at']): d for d in list_of_dicts}

Since you mention no preference in your question, this will probably take the last duplicate.

You could create your own dict object with a special hash, but this seems easier. If you want a list back then just take filtered_dict.values().

If by chance you only want the first match you are going to have to add a few lines of code.:

existing_dicts = set()
filtered_list = []
for d in list_of_dicts:
    if (d['id'], d['updated_at']) notin existing_dicts:
        existing_dicts.add((d['id'], d['updated_at']))
        filtered_list.append(d)