How to Remove Duplicates from a List of Python Dictionaries by Key

To remove duplicates from a list of Python dictionaries by a single key, it is usually enough to iterate over the list once, store key values that have already been seen in a set, and add only the first dictionary with each new value to the result. This approach keeps the first matching item for each key value and does not modify the original list.

Basic solution: keep the first dictionary for each key value

Suppose you have a list of users where the id field should be unique:

items = [
    {"id": 1, "name": "Alice"},
    {"id": 2, "name": "Bob"},
    {"id": 1, "name": "Alice updated"},
    {"id": 3, "name": "Carol"},
    {"id": 2, "name": "Bob updated"},
]

Remove duplicates by id:

seen = set()
unique_items = []
for item in items:
key = item["id"]
if key not in seen:    seen.add(key)    unique_items.append(item)

The unique_items list will contain the dictionaries with the first occurrence of each id:

[
    {"id": 1, "name": "Alice"},
    {"id": 2, "name": "Bob"},
    {"id": 3, "name": "Carol"},
]

You can verify the result with a regular assert:

assert unique_items == [
    {"id": 1, "name": "Alice"},
    {"id": 2, "name": "Bob"},
    {"id": 3, "name": "Carol"},
]

Why use a set

The seen set stores key values that have already occurred. For each dictionary, three actions are performed:

get the value of the required field;
check whether it has been seen before;
if not, store the value in seen and add the dictionary itself to the result list.

You do not need to put the dictionaries themselves into the set: a dictionary is mutable and cannot be used as an element of a set. Only the field value used to determine uniqueness is stored in the set.

The key value added to a set must be hashable. Strings, numbers, None, and tuples containing hashable objects are suitable. Lists and dictionaries are not.

A reusable function

If you need this operation in several places, it is more convenient to move it into a function:

def unique_by_key(items, key):
    seen = set()
    result = []
for item in items:    value = item[key]    if value not in seen:        seen.add(value)        result.append(item)return result

Usage:

users = unique_by_key(items, "id")

Verification:

assert [item["id"] for item in users] == [1, 2, 3]

The function assumes that every dictionary contains the specified key. If even one item does not contain it, the expression item[key] will raise a KeyError. This is useful behavior when a missing field indicates invalid input data.

What to do if the key may be missing

If incomplete dictionaries are allowed, you first need to define how a missing field should be handled. For example, you can skip such items:

def unique_by_existing_key(items, key):
    seen = set()
    result = []
for item in items:    if key not in item:        continue    value = item[key]    if value not in seen:        seen.add(value)        result.append(item)return result

Another option is to treat a missing key as a separate value. You can use a unique marker object for this:

missing = object()
seen = set()
result = []
for item in items:
value = item.get("id", missing)
if value not in seen:    seen.add(value)    result.append(item)

In this case, only the first dictionary without an id will be kept. Using None as the marker is not always correct because None may be a valid real value for the field.

If you need to keep the last occurrence

Sometimes a newer dictionary should replace an older one. For example, a later record with the same id may contain updated data. One clear approach is to store the index of the item that has already been added:

def unique_by_key_keep_last(items, key):
    indexes = {}
    result = []
for item in items:    value = item[key]    if value in indexes:        result[indexes[value]] = item    else:        indexes[value] = len(result)        result.append(item)return result

For the original example, the result will be equivalent to:

[
    {"id": 1, "name": "Alice updated"},
    {"id": 2, "name": "Bob updated"},
    {"id": 3, "name": "Carol"},
]

Verification:

result = unique_by_key_keep_last(items, "id")
assert result == [
{"id": 1, "name": "Alice updated"},
{"id": 2, "name": "Bob updated"},
{"id": 3, "name": "Carol"},
]

With this approach, an item's position is determined by its first occurrence, while its contents are replaced by later matches.

Short version using a dictionary

If you need to keep the last value for each key and do not need any additional duplicate-handling logic, you can build an auxiliary dictionary:

unique_items = list({
    item["id"]: item
    for item in items
}.values())

When the same dictionary key is assigned again, its value is replaced, so the last corresponding object for each repeated id remains. This approach is shorter, but an explicit loop is usually more convenient if you also need to validate data, count duplicates, or choose between the first and last item based on an additional condition.

Removing duplicates by multiple fields

If uniqueness is defined by a combination of values rather than a single field, you can store a tuple in the set. For example, suppose a record is considered a duplicate only when both first_name and last_name match:

people = [
    {"first_name": "Ivan", "last_name": "Petrov", "age": 30},
    {"first_name": "Anna", "last_name": "Ivanova", "age": 25},
    {"first_name": "Ivan", "last_name": "Petrov", "age": 31},
]
seen = set()
unique_people = []
for person in people:
key = (person["first_name"], person["last_name"])
if key not in seen:    seen.add(key)    unique_people.append(person)

The result will contain the first record for each unique first-name and last-name pair.

If the key value is a list or dictionary

The following code will fail if item["tags"] contains a list:

seen.add(item["tags"])

The reason is that a list is not hashable. If the order of the list elements matters and the elements themselves are hashable, you can convert the list to a tuple:

key = tuple(item["tags"])

For example:

items = [
    {"id": 1, "tags": ["python", "api"]},
    {"id": 2, "tags": ["python", "api"]},
    {"id": 3, "tags": ["api", "python"]},
]
seen = set()
result = []
for item in items:
key = tuple(item["tags"])
if key not in seen:    seen.add(key)    result.append(item)

Here, the lists ["python", "api"] and ["api", "python"] are considered different. If tag order should not affect uniqueness, you need to normalize the data separately according to the requirements of the task. Simple sorting is suitable only when the elements can actually be compared that way and changing their order does not alter the meaning of the data.

Do not confuse duplicates by key with full dictionary equality

The task of “removing identical dictionaries” is different from the task of “keeping one dictionary for each id.” For example:

{"id": 1, "name": "Alice"}
{"id": 1, "name": "Alice updated"}

The dictionaries are different, but they are duplicates by the id field. Therefore, you first need to define the uniqueness criterion and then build the key specifically from that criterion.

How to count removed duplicates

If you also need to know the number of duplicate records, you can add a counter in the same pass:

seen = set()
result = []
duplicates = 0
for item in items:
value = item["id"]
if value in seen:    duplicates += 1    continueseen.add(value)result.append(item)

You can also verify the number of removed items after processing:

assert duplicates == len(items) - len(result)

This approach is useful when processing imported data: the result contains unique records, while the counter lets you separately track how many duplicates were discarded.

Common mistakes

Adding the dictionary itself to a set. A regular dict cannot be used as an element of a set.
Using item.get without choosing the default value carefully. A missing key and a real None value may accidentally be treated as the same group.
Not deciding which duplicate to keep. Some tasks require the first item, while others require the last one.
Using an unhashable value as the uniqueness key. A list or dictionary must first be represented in a suitable immutable form if doing so is valid for the meaning of the data.
Removing items from the original list while iterating over it directly. It is simpler and safer to build a new result list.

Final checklist

Define the field or set of fields that determines whether records are duplicates.
Decide in advance whether to keep the first or last occurrence.
To keep the first occurrence, use a set of keys that have already been seen and a separate result list.
Make sure the value stored in the set is hashable.
Define how to handle dictionaries without the required key: raise an error, skip them, or treat them as a separate group.
For multiple fields, use a tuple such as (item["a"], item["b"]).
Verify the result with an assert on a small dataset containing both unique records and duplicates.