Collectives™ on Stack Overflow

Find centralized, trusted content and collaborate around the technologies you use most.

Teams

Q&A for work

Connect and share knowledge within a single location that is structured and easy to search.

Modified versions of http://www.peterbe.com/plog/uniqifiers-benchmark

To preserve the order:

def f(seq): # Order preserving
  ''' Modified version of Dave Kirby solution '''
  seen = set()
  return [x for x in seq if x not in seen and not seen.add(x)]
OK, now how does it work, because it's a little bit tricky here if x not in seen and not seen.add(x):
In [1]: 0 not in [1,2,3] and not print('add')
Out[1]: True
Why does it return True? print (and set.add) returns nothing:
In [3]: type(seen.add(10))
Out[3]: <type 'NoneType'>
and not None == True, but:
In [2]: 1 not in [1,2,3] and not print('add')
Out[2]: False
Why does it print 'add' in [1] but not in [2]? See False and print('add'), and doesn't check the second argument, because it already knows the answer, and returns true only if both arguments are True.
More generic version, more readable, generator based, adds the ability to transform values with a function:
def f(seq, idfun=None): # Order preserving
  return list(_f(seq, idfun))
def _f(seq, idfun=None):  
  ''' Originally proposed by Andrew Dalke '''
  seen = set()
  if idfun is None:
    for x in seq:
      if x not in seen:
        seen.add(x)
        yield x
  else:
    for x in seq:
      x = idfun(x)
      if x not in seen:
        seen.add(x)
        yield x
Without order (it's faster):
def f(seq): # Not order preserving
  return list(set(seq))
                sort of inner helper function (there was a bug in the code, should be _f instead of _f10 on line 2, thanks for spotting)
– Paweł Prażak
                Apr 10, 2011 at 6:45
                @DannyStaple: that works in python 2, but in python 3 it returns a view of the dictionary keys, which might be okay for some purposes, but doesn't support indexing for example.
– Mark
                Dec 3, 2016 at 11:57
                The initial one liner will work. The aternative form returns an odict_keys type, which is less useful for this - but can still be converted to a list.
– Danny Staple
                Dec 4, 2016 at 0:19
if you have Python list
>>> randomList = ["a","f", "b", "c", "d", "a", "c", "e", "d", "f", "e"]
and you want to remove duplicates from it.
>>> uniqueList = []
>>> for letter in randomList:
    if letter not in uniqueList:
        uniqueList.append(letter)
>>> uniqueList
['a', 'f', 'b', 'c', 'd', 'e']
This is how you can remove duplicates from the list.
                +1 because it's the only one that works for types that are unhashable, but do have an eq function (if your types are hashable, use one of the other solutions). Note that it will be slow for very big lists.
– Claude
                Sep 9, 2014 at 19:48
                Unless in some special case as Claude explained, this one has the worst performance: O(n^2)
– Yingbo Miao
                Jun 18, 2020 at 20:26
                map just creates map object (generator), does not execute it. list(map(....)) forces the execution
– user2389519
                Jul 4, 2022 at 6:53
To @Danny's comment: my original suggestion does not keep the keys ordered. If you need the keys sorted, try:

>>> from collections import OrderedDict
>>> OrderedDict( (x,1) for x in mylist ).keys()
[3, 2, 1, 4, 5]
which keeps elements in the order by the first occurrence of the element (not extensively tested)
                This would not preserve order - dictionary order (and set order) is determined by the hashing algorithm and not insertion order. I am not sure of the effects of a dictionary comprehension with an OrderedDict type though.
– Danny Staple
                Jan 28, 2015 at 14:00
                @DannyStaple True. I added an example using OrderedDict and a generator, if ordered output is desired.
– cod3monk3y
                Jan 28, 2015 at 17:03
The characteristics of sets in Python are that the data items in a set
are unordered and duplicates are not allowed. If you try to add a data item to a set that already contains the data item, Python simply ignores it.
>>> l = ['a', 'a', 'bb', 'b', 'c', 'c', '10', '10', '8','8', 10, 10, 6, 10, 11.2, 11.2, 11, 11]
>>> distinct_l = set(l)
>>> print(distinct_l)
set(['a', '10', 'c', 'b', 6, 'bb', 10, 11, 11.2, '8'])
If all elements of the list may be used as dictionary keys (i.e. they are all hashable) this is often faster. Python Programming FAQ
d = {}
for x in mylist:
    d[x] = 1
mylist = list(d.keys())
                In Python, sets and dicts are built using hashtables so they are interchangeable in this scenario.  They both provide the same operations (limiting duplicates) and both have the same running time.
– brildum
                Dec 16, 2010 at 14:29
The simplest way to remove duplicates whilst preserving order is to use collections.OrderedDict (Python 2.7+).
from collections import OrderedDict
d = OrderedDict()
for x in mylist:
    d[x] = True
print d.iterkeys()
				I want to compare each value from left to right in a text file and find unique values they all are in new line
                See more linked questions