How to Find Duplicates in a Python List While Preserving Order

What “finding duplicates while preserving order” means

In Python, finding duplicates can refer to different results. For the list [4, 2, 4, 3, 2, 4], there are at least three possible variants:

• [4, 2] — each duplicated value is returned once, in the order its first duplicate is detected;
• [4, 2] — values are ordered by their first appearance in the original list;
• [4, 2, 4] — all repeated occurrences are preserved except the first occurrence of each value.

For this example, the first two results are identical, but that is not always the case. For example, with the list ["a", "b", "b", "a"], ordering by the first duplicate produces ["b", "a"], while ordering by the first appearance produces ["a", "b"].

Unique duplicates in the order they are detected

The most practical approach is to iterate over the list once and use two sets:

• seen stores elements that have already been encountered;
• added prevents the same duplicate from being added more than once.

def find_duplicates(items):
    seen = set()
    added = set()
    duplicates = []
for item in items:
    if item in seen and item not in added:
        duplicates.append(item)
        added.add(item)
    else:
        seen.add(item)

return duplicates
values = [4, 2, 4, 3, 2, 4]
print(find_duplicates(values))
[4, 2]

The result order is determined by when each value is encountered for the second time. The value 4 is repeated before 2, so it appears first.

The average time complexity of this solution is O(n), where n is the length of the list. Additional memory usage can also reach O(n).

Sets can only be used with hashable elements: numbers, strings, tuples containing hashable values, and some other immutable objects. Lists and dictionaries cannot be added directly to a set.

Duplicates in the order of their first appearance

Sometimes you need to determine the frequency of each value first, then iterate over the original list and select elements that occur more than once. collections.Counter is convenient for counting.

from collections import Counter
def find_duplicates_by_first_position(items):
counts = Counter(items)
result = []
added = set()
for item in items:
    if counts[item] > 1 and item not in added:
        result.append(item)
        added.add(item)

return result
values = ["a", "b", "b", "a"]
print(find_duplicates_by_first_position(values))
['a', 'b']

Here, the string "a" appears first in the result because it occurred earlier in the original list, even though its duplicate is detected after the duplicate of "b".

This approach also runs in O(n) average time, but it performs two logical passes: the first counts the elements, and the second builds the result.

All repeated occurrences except the first instances

When every repeated occurrence must be preserved, one set is enough. The first instance is added to seen, and all subsequent instances are added to the result.

def find_repeated_occurrences(items):
    seen = set()
    repeated = []
for item in items:
    if item in seen:
        repeated.append(item)
    else:
        seen.add(item)

return repeated
values = [4, 2, 4, 3, 2, 4]
print(find_repeated_occurrences(values))
[4, 2, 4]

This variant is useful when analyzing sequences of events, identifiers, or log entries where both the presence of duplicates and the number of extra occurrences matter.

Getting values together with duplicate positions

When diagnosing data, knowing only the duplicated value is often not enough. It is also useful to obtain the indexes at which it was repeated.

def find_duplicate_positions(items):
    first_positions = {}
    duplicates = []
for index, item in enumerate(items):
    if item in first_positions:
        duplicates.append({
            "value": item,
            "first_index": first_positions[item],
            "duplicate_index": index,
        })
    else:
        first_positions[item] = index

return duplicates
values = ["red", "blue", "red", "green", "blue"]
for duplicate in find_duplicate_positions(values):
print(duplicate)

Result:

{'value': 'red', 'first_index': 0, 'duplicate_index': 2}
{'value': 'blue', 'first_index': 1, 'duplicate_index': 4}

The function preserves the order of repeated occurrences and shows where each value first appeared. If an element is repeated several times, every additional position is returned separately.

A generator for large sequences

For a large list or a gradually arriving data stream, there is no need to create the complete result list immediately. A generator returns duplicates one at a time.

def iter_duplicates(items):
    seen = set()
    yielded = set()
for item in items:
    if item in seen:
        if item not in yielded:
            yielded.add(item)
            yield item
    else:
        seen.add(item)
values = [10, 20, 10, 30, 20, 40]
for duplicate in iter_duplicates(values):
print(duplicate)

The generator does not store a separate list of found duplicates, but the seen and yielded sets still consume memory. This approach is useful when processing can begin immediately without waiting for the entire input sequence to be traversed.

Working with lists and dictionaries

Elements such as [1, 2] or {"id": 5} are unhashable. Attempting to add them to a set raises a TypeError.

values = [[1, 2], [3, 4], [1, 2]]
seen = set()
seen.add(values[0])
TypeError: unhashable type: 'list'

For nested lists, each element can be converted to a tuple:

def find_duplicate_lists(items):
    seen = set()
    added = set()
    duplicates = []
for item in items:
    key = tuple(item)

    if key in seen and key not in added:
        duplicates.append(item)
        added.add(key)
    else:
        seen.add(key)

return duplicates
values = [[1, 2], [3, 4], [1, 2], [1, 2]]
print(find_duplicate_lists(values))
[[1, 2]]

For dictionaries, a key can be constructed from sorted key-value pairs:

def find_duplicate_dicts(items):
    seen = set()
    added = set()
    duplicates = []
for item in items:
    key = tuple(sorted(item.items()))

    if key in seen and key not in added:
        duplicates.append(item)
        added.add(key)
    else:
        seen.add(key)

return duplicates
values = [
{"id": 1, "name": "Ann"},
{"id": 2, "name": "Bob"},
{"name": "Ann", "id": 1},
]
print(find_duplicate_dicts(values))
[{'name': 'Ann', 'id': 1}]

Conversion with tuple(sorted(item.items())) is suitable only when the dictionary keys and values can be compared and hashed correctly. Nested dictionaries and lists require recursive data normalization.

A generic variant with a key function

In real-world tasks, duplicates are often identified by one field rather than by the entire object. For example, user records may be considered identical when they have the same identifier.

def find_duplicates_by(items, key):
    seen = set()
    added = set()
    duplicates = []
for item in items:
    marker = key(item)

    if marker in seen and marker not in added:
        duplicates.append(item)
        added.add(marker)
    else:
        seen.add(marker)

return duplicates
users = [
{"id": 10, "name": "Anna"},
{"id": 20, "name": "Boris"},
{"id": 10, "name": "Anna Updated"},
{"id": 30, "name": "Chris"},
{"id": 20, "name": "Boris Updated"},
]
duplicates = find_duplicates_by(users, key=lambda user: user["id"])
print(duplicates)
[{'id': 10, 'name': 'Anna Updated'},
{'id': 20, 'name': 'Boris Updated'}]

The result contains the object at which the corresponding identifier is repeated for the first time. The order of duplicates is preserved.

A compact list-comprehension form

The task can be written compactly, but overly short constructs are usually harder to read. One possible variant for obtaining all repeated occurrences is:

seen = set()
values = [1, 2, 1, 3, 2, 1]
duplicates = [
item
for item in values
if item in seen or not seen.add(item)
]
print(duplicates)
[1, 2, 1]

This form is best avoided. The set.add() method returns None, and the condition relies on a side effect inside the list comprehension. The code is not self-explanatory and is harder to maintain.

A clearer function with a regular loop is almost always preferable:

def find_repeated_occurrences(items):
    seen = set()
    result = []
for item in items:
    if item in seen:
        result.append(item)
    else:
        seen.add(item)

return result

Why you should not use count inside a loop

A common construct looks simple:

values = [1, 2, 1, 3, 2]
duplicates = []
for item in values:
if values.count(item) > 1 and item not in duplicates:
duplicates.append(item)

The list.count() method traverses the entire list each time it is called. Calling it for every element can increase the overall complexity to O(n²). The difference is negligible for small datasets, but with tens or hundreds of thousands of elements, this code becomes substantially slower than a set-based solution.

The additional item not in duplicates check is also linear because duplicates is a list. A separate set is better for ensuring uniqueness in the result.

Comparison of the main approaches

A ready-to-use function for most tasks

The following implementation is suitable for a regular list of hashable elements:

def find_duplicates(items):
    """Returns unique duplicates in the order of their first repetition."""
    seen = set()
    duplicates = []
    duplicate_keys = set()
for item in items:
    if item in seen:
        if item not in duplicate_keys:
            duplicates.append(item)
            duplicate_keys.add(item)
    else:
        seen.add(item)

return duplicates

Test examples:

assert find_duplicates([]) == []
assert find_duplicates([1]) == []
assert find_duplicates([1, 1]) == [1]
assert find_duplicates([1, 2, 1, 2]) == [1, 2]
assert find_duplicates([1, 1, 1]) == [1]
assert find_duplicates(["b", "a", "a", "b"]) == ["a", "b"]

Final checklist

• Determine whether you need the order of first appearance or first repetition.
• Use a set for hashable elements and an average complexity of O(n).
• Use two sets when each duplicate should appear in the result only once.
• Use one set when all repeated occurrences are required.
• Use Counter when frequencies and the order of first appearance matter.
• For dictionaries and objects, construct a hashable key or pass a key function.
• Do not call list.count() inside a loop for large lists.
• Add tests for an empty list, a list without duplicates, and a value repeated several times.