How to Find Duplicates in a Python List While Preserving Order

To find duplicates in a Python list while preserving the order of their first repeated occurrence, scan the list from left to right, store values already seen in one set, and store duplicates already added to the result in another. This approach does not sort the source data or modify the list.

items = [4, 2, 7, 2, 4, 2, 9, 7]
seen = set()
added = set()
duplicates = []
for item in items:
if item in seen:
if item not in added:
duplicates.append(item)
added.add(item)
else:
seen.add(item)
print(duplicates)
[2, 4, 7]

The result [2, 4, 7] matches the order in which the values first become duplicates: the second occurrence of 2 appears before the second occurrence of 4, followed by the repeated 7.

Ready-to-use function

For reuse, move the algorithm into a function:

def find_duplicates(items):
    seen = set()
    added = set()
    duplicates = []
for item in items:
    if item in seen:
        if item not in added:
            duplicates.append(item)
            added.add(item)
    else:
        seen.add(item)

return duplicates

You can test the function with several inputs:

print(find_duplicates([1, 3, 1, 2, 3]))
# [1, 3]
print(find_duplicates(["a", "b", "a", "c", "b"]))
['a', 'b']
print(find_duplicates([1, 2, 3]))
[]
print(find_duplicates([]))
[]

The function returns each duplicated value only once. The original list remains unchanged.

How the algorithm works step by step

Consider this list:

items = [5, 3, 5, 8, 3, 5]

1. The first value, 5, has not been seen yet. It is added to seen.
2. The value 3 also appears for the first time and is added to seen.
3. The next 5 is already in seen. It is a duplicate, so 5 is added to duplicates and to added.
4. The value 8 appears for the first time and is added only to seen.
5. The second 3 is found in seen but is not in added. Therefore, 3 is added to the result.
6. The final 5 is already present in both seen and added. It is not added to the result again.

Result:

[5, 3]

Why one set is not enough

If you only need to know whether duplicates exist, a set of values already seen is enough. But to produce a list of unique duplicated values, you also need to remember which duplicates have already been added to the result.

For example, this code adds the same value multiple times:

items = [1, 1, 1, 1]
seen = set()
duplicates = []
for item in items:
if item in seen:
duplicates.append(item)
else:
seen.add(item)
print(duplicates)
[1, 1, 1]

If the required result is [1], you need the second added set or another way to check whether the result already contains that value.

Shorter version

The same algorithm can be written a little more compactly without changing its behavior:

def find_duplicates(items):
    seen = set()
    duplicates = []
    added = set()
for x in items:
    if x in seen and x not in added:
        duplicates.append(x)
        added.add(x)
    seen.add(x)

return duplicates

Here, seen.add(x) is called on every iteration. Adding an element that is already present in a set does not create a second copy.

The set-based method works only with hashable elements. Strings, numbers, and tuples containing hashable values generally work. Lists and dictionaries cannot be added directly to a set, so they require a different approach.

If the list elements are themselves lists

For nested lists, the following approach will not work:

items = [[1, 2], [3, 4], [1, 2]]
seen = set()
for item in items:
seen.add(item)  # TypeError: list is unhashable

If nested lists can be represented unambiguously as tuples, use the tuple as the lookup key while keeping the original value in the result:

def find_duplicate_lists(items):
    seen = set()
    added = set()
    duplicates = []
for item in items:
    key = tuple(item)

    if key in seen and key not in added:
        duplicates.append(item)
        added.add(key)

    seen.add(key)

return duplicates
items = [[1, 2], [3, 4], [1, 2], [5], [3, 4]]
print(find_duplicate_lists(items))
[[1, 2], [3, 4]]

This conversion is valid only when the contents of the nested list can also form a hashable tuple. For example, a tuple containing a regular list still cannot be used as an element of a set.

General approach for unhashable values

If the elements cannot be converted into a reliable hashable key, you can find matches using ordinary equality comparisons:

def find_duplicates_unhashable(items):
    seen = []
    duplicates = []
for item in items:
    if item in seen:
        if item not in duplicates:
            duplicates.append(item)
    else:
        seen.append(item)

return duplicates

Example:

items = [
    {"id": 1},
    {"id": 2},
    {"id": 1},
    {"id": 3},
    {"id": 2},
]
print(find_duplicates_unhashable(items))
[{'id': 1}, {'id': 2}]

The order is preserved, but this approach can be significantly slower than using sets: item in seen performs a sequential search through the list. As the number of elements grows, the number of comparisons can grow quadratically.

If you need every repeated occurrence

Sometimes “find duplicates” means returning every occurrence after the first one rather than returning each duplicated value only once. In that case, the second added set is unnecessary:

def find_repeated_occurrences(items):
    seen = set()
    duplicates = []
for item in items:
    if item in seen:
        duplicates.append(item)
    else:
        seen.add(item)

return duplicates
print(find_repeated_occurrences([2, 5, 2, 2, 5]))
[2, 2, 5]

Here, the two later instances of 2 are treated as two separate repeated occurrences.

If you need values that occur more than once

There is another possible interpretation of order: return duplicated values according to their first appearance in the original list rather than according to when their second occurrence is detected.

The difference is visible in this list:

items = ["a", "b", "b", "a"]

The algorithm from the beginning of the article returns ["b", "a"] because the second "b" appears before the second "a". If you need ["a", "b"], first count the elements and then make another pass through the original list.

For hashable values, this can be done without sorting:

def find_duplicates_by_first_appearance(items):
    counts = {}
for item in items:
    counts[item] = counts.get(item, 0) + 1

result = []
added = set()

for item in items:
    if counts[item] > 1 and item not in added:
        result.append(item)
        added.add(item)

return result
print(find_duplicates_by_first_appearance(
["a", "b", "b", "a"]
))
['a', 'b']

Here, the result order is determined directly by the second pass through the original list. The algorithm does not rely on the iteration order of the counts dictionary itself.

How to choose an approach

Complexity

For hashable elements, the set-based approach is usually preferable because set membership checks are designed for efficient hash-based lookup. The algorithm also stores a set of previously seen elements, a set of duplicates already found, and the result list.

You should not promise a fixed execution time for a particular call: it depends on the amount of data, object types, their hashing and comparison methods, the Python implementation, and the runtime environment.

The list-based approach for unhashable objects uses sequential in checks. In the worst case, the number of comparisons grows as O(n²), so for large datasets it is better to define a stable hashable key when the data structure allows it.

Checking the result

A minimal set of checks for a function that returns each duplicate only once:

assert find_duplicates([1, 2, 1]) == [1]
assert find_duplicates([1, 2, 1, 2]) == [1, 2]
assert find_duplicates([1, 1, 1]) == [1]
assert find_duplicates([1, 2, 3]) == []
assert find_duplicates([]) == []
assert find_duplicates(["x", "y", "x"]) == ["x"]

If these expressions complete without an AssertionError, the function returns the expected results for these specific test cases. This does not prove correctness for every possible custom object type, especially objects with nonstandard equality or hashing implementations.

Final checklist

• Need to preserve the order in which duplicates are detected — scan the original list from left to right.
• Need to return each duplicate only once — use seen and added.
• Need every repeated occurrence — seen alone is enough.
• Need the order of the values' first appearance — count frequencies and make a second pass through the original list.
• Before using set, make sure the elements are hashable.
• For lists and dictionaries, define a hashable key or use comparison without a set.
• Do not sort the data if you need to preserve the original order.
• Test an empty list, a list with no duplicates, repeated duplicates, and the actual data types used by the application separately.