In this article, we’ll look at how we can remove duplicate elements from List in Python. There are multiple ways of approaching this problem, and we will show you some of them.
Methods to Remove Duplicate Elements from List – Python
1. Using iteration
To remove duplicate elements from List in Python, we can manually iterate through the list and add an element to the new list if it is not present. Otherwise, we skip that element.
The code is shown below:
a = [2, 3, 3, 2, 5, 4, 4, 6] b =  for i in a: # Add to the new list # only if not present if i not in b: b.append(i) print(b)
[2, 3, 5, 4, 6]
The same code can be written using List Comprehension to reduce the number of lines of code, although it is essentially the same as before.
a = [2 3, 4, 2, 5, 4, 4, 6] b =  [b.append(i) for i in a if i not in b] print(b)
The problem with this approach is that it is a bit slow since a comparison is done for every element in the new list, while already iterating through our original list.
This is computationally expensive, and we have other methods to deal with this issue. You should use this only if the list size is not very large. Otherwise, refer to the other methods.
2. Using set()
A simple and fast approach to remove duplicate elements from list in Python would be to use Python’s built-in
set() method to convert the list elements into a unique set, following which we can convert it into a List now removed of all its duplicate elements.
first_list = [1, 2, 2, 3, 3, 3, 4, 5, 5, 6] # Convert to a set first set_list = set(first_list) # Now convert the set into a List print(list(set_list)) second_list = [2, 3, 3, 2, 5, 4, 4, 6] # Does the same as above, in a single line print(list(set(second_list)))
[1, 2, 3, 4, 5, 6] [2, 3, 4, 5, 6]
The problem with this approach is that the original List order is not maintained as with the case of the second List since we create the new List from an unordered Set. so if you wish to still preserve the relative ordering, you must avoid this method.
3. Preserving Order: Use OrderedDict
If you want to preserve the order while you remove duplicate elements from List in Python, you can use the OrderedDict class from the collections module.
More specifically, we can use
OrderedDict.fromkeys(list) to obtain a dictionary having duplicate elements removed, while still maintaining order. We can then easily convert it into a list using the
from collections import OrderedDict a = [2, 3, 3, 2, 5, 4, 4, 6] b = list(OrderedDict.fromkeys(a)) print(b)
[2, 3, 5, 4, 6]
NOTE: If you have Python 3.7 or later, we can use the built in
dict.fromkeys(list) instead. This will also guarantee the order.
As you can observe, the order is indeed maintained, so we get the same output as of the first method. But this is much faster! This is the recommended solution to this problem. But for illustration, we will show you a couple of more approaches to remove duplicate elements from List in Python.
4. Using list.count()
list.count() method returns the number of occurrences of the value. We can use it along with the
remove() method to eliminate any duplicate elements. But again, this does not preserve the order.
Note that this method modifies the input list in place, so the changes are reflected there itself.
a = [0, 1, 2, 3, 4, 1, 2, 3, 5] for i in a: if a.count(i) > 1: a.remove(i) print(a)
[0, 4, 1, 2, 3, 5]
Everything seems fine, isn’t it?
But, there is a small issue with the above code.
When we are iterating over the list using the for loop and removing the element at the same time, the iterator skips one element. So, the code output depends on the list elements and if you are lucky then you will never get the issue. Let’s understand this scenario with a simple code.
a = [1, 2, 3, 2, 5] for i in a: if a.count(i) > 1: a.remove(i) print(a, i) print(a)
[1, 2, 3, 2, 5] 1 [1, 3, 2, 5] 2 [1, 3, 2, 5] 2 [1, 3, 2, 5] 5 [1, 3, 2, 5]
You can see that the for loop is executed only four times and it’s skipping 3, the next element after the remove() call. If you pass the input list as [1, 1, 1, 1], the final list will be [1, 1].
So, is there any workaround?
Of course, there is a workaround. Use the copy of the list in the for loop but remove the elements from the main list. A simple way to create a copy of the list is through slicing. Here is the update code that will work fine in all the cases.
a = [1, 1, 1, 1] for i in a[:]: # using list copy for iteration if a.count(i) > 1: a.remove(i) print(a, i) print(a)
[1, 1, 1] 1 [1, 1] 1  1  1 
5. Using sort()
We can use the
sort() method to sort the set that we obtained in approach 2. This will also remove any duplicates, while preserving the order, but is slower than the
a = [0, 1, 2, 3, 4, 1, 2, 3, 5] b = list(set(a)) b.sort(key=a.index) print(b)
[0, 1, 2, 3, 4, 5]
6. Using pandas module
In case we are working with the Pandas module, we can use the
pandas.drop_duplicates() method to remove the duplicates and then convert it into a List, while also preserving the order.
import pandas as pd a = [0, 1, 2, 3, 4, 1, 2, 3, 5] pd.Series(a).drop_duplicates().tolist()
[0, 1, 2, 3, 4, 5]
- JournalDev article on Removing Duplicate List Elements
- StackOverflow Question