I haven't been able to find an understandable explanation of how to actually use Python's itertools.groupby() function. What I'm trying to do is this:

  • Take a list - in this case, the children of an objectified lxml element
  • Divide it into groups based on some criteria
  • Then later iterate over each of these groups separately.

I've reviewed the documentation, and the examples, but I've had trouble trying to apply them beyond a simple list of numbers.

So, how do I use of itertools.groupby()? Is there another technique I should be using? Pointers to good "prerequisite" reading would also be appreciated.

12 Answers 12


IMPORTANT NOTE: You have to sort your data first.

The part I didn't get is that in the example construction

groups = []
uniquekeys = []
for k, g in groupby(data, keyfunc):
   groups.append(list(g))    # Store group iterator as a list

k is the current grouping key, and g is an iterator that you can use to iterate over the group defined by that grouping key. In other words, the groupby iterator itself returns iterators.

Here's an example of that, using clearer variable names:

from itertools import groupby

things = [("animal", "bear"), ("animal", "duck"), ("plant", "cactus"), ("vehicle", "speed boat"), ("vehicle", "school bus")]

for key, group in groupby(things, lambda x: x[0]):
    for thing in group:
        print "A %s is a %s." % (thing[1], key)
    print " "

This will give you the output:

A bear is a animal.
A duck is a animal.

A cactus is a plant.

A speed boat is a vehicle.
A school bus is a vehicle.

In this example, things is a list of tuples where the first item in each tuple is the group the second item belongs to.

The groupby() function takes two arguments: (1) the data to group and (2) the function to group it with.

Here, lambda x: x[0] tells groupby() to use the first item in each tuple as the grouping key.

In the above for statement, groupby returns three (key, group iterator) pairs - once for each unique key. You can use the returned iterator to iterate over each individual item in that group.

Here's a slightly different example with the same data, using a list comprehension:

    listOfThings = " and ".join([thing[1] for thing in group])
    print key + "s:  " + listOfThings + "."

This will give you the output:

animals: bear and duck.
plants: cactus.
vehicles: speed boat and school bus.

  • 1
    Is there a way to specify the groups beforehand and then not require sorting? – John Salvatier May 10 '11 at 19:39
  • 3
    @Julian python docs seem great for most stuff but when it comes to iterators, generators, and cherrypy the docs mostly mystify me. Django's docs are doubly baffling. – Marc Maxmeister Oct 1 '12 at 18:19
  • 1
    You can use a single list comprehension to get lists of groups: item_groups = [group[1] for group in itertools.groupby(items, lambda item: item.property)] or if they are sorted: group_dict = dict([(key, group) for key, group in itertools.groupby(items, lambda item: item.property)]) – Danny Staple Apr 10 '14 at 16:28
  • 4
    +1 for the sorting -- I didn't understand what you meant until I grouped my data. – Cody Apr 24 '14 at 2:25
  • 2
    @DavidCrook very late to the party but might help someone. It's probably because your array is not sorted try groupby(sorted(my_collection, key=lambda x: x[0]), lambda x: x[0])) under the assumption that my_collection = [("animal", "bear"), ("plant", "cactus"), ("animal", "duck")] and you want to group by animal or plant – Robin Nemeth Dec 14 '17 at 15:12

Can you show us your code?

The example on the Python docs is quite straightforward:

groups = []
uniquekeys = []
for k, g in groupby(data, keyfunc):

So in your case, data is a list of nodes, keyfunc is where the logic of your criteria function goes and then groupby() groups the data.

You must be careful to sort the data by the criteria before you call groupby or it won't work. groupby method actually just iterates through a list and whenever the key changes it creates a new group.

  • 27
    So you read keyfunc and were like "yeah, I know exactly what that is because this documentation is quite straightforward."? Incredible! – Jarad Apr 7 '17 at 19:22
  • 3
    I believe most people know already about this "straightforward" but useless example, since it doesn't say what kind of 'data' and 'keyfunc' to use!! But I guess you don't know either, otherwise you would help people by clarifying it and not just copy-pasting it. Or do you? – Apostolos Mar 28 '18 at 19:14

itertools.groupby is a tool for grouping items.

From the docs, we glean further what it might do:


# [list(g) for k, g in groupby('AAAABBBCCD')] --> AAAA BBB CC D

groupby objects yield key-group pairs where the group is a generator.


  • A. Group consecutive items together
  • B. Group all occurrences of an item, given a sorted iterable
  • C. Specify how to group items with a key function


# Define a printer for comparing outputs
>>> def print_groupby(iterable, key=None):
...    for k, g in it.groupby(iterable, key):
...        print("key: '{}'--> group: {}".format(k, list(g)))

# Feature A: group consecutive occurrences
>>> print_groupby("BCAACACAADBBB")
key: 'B'--> group: ['B']
key: 'C'--> group: ['C']
key: 'A'--> group: ['A', 'A']
key: 'A'--> group: ['A']
key: 'D'--> group: ['D']
key: 'B'--> group: ['B', 'B', 'B']

# Feature B: group all occurrences
>>> print_groupby(sorted("BCAACACAADBBB"))
key: 'A'--> group: ['A', 'A', 'A', 'A', 'A']
key: 'C'--> group: ['C', 'C', 'C']

# Feature C: group by a key function
>>> key = lambda x: x.islower()
>>> print_groupby(sorted("bCAaCacAADBbB"), key)
key: 'True'--> group: ['a', 'a', 'b', 'b', 'c']


Note: Several of the latter examples derive from Víctor Terrón's PyCon (talk) (Spanish), "Kung Fu at Dawn with Itertools". See also the groupbysource code written in C.


# OP: Yes, you can use `groupby`, e.g. 
[do_something(list(g)) for _, g in groupby(lxml_elements, key=criteria_func)]
  • Technically, the docs should probably say [''.join(g) for k, g in groupby('AAAABBBCCD')] --> AAAA BBB CC D. – Mateen Ulhaq Oct 24 '18 at 22:55
  • 1
    Yes. Most of the itertools docstrings are "abridged" in this way. Since all of the itertools are iterators, they must be cast to a builtin (list(), tuple()) or consumed in a loop/comprehension to display the contents. These are redundancies the author likely excluded to conserve space. – pylang Oct 25 '18 at 0:13

A neato trick with groupby is to run length encoding in one line:

[(c,len(list(cgen))) for c,cgen in groupby(some_string)]

will give you a list of 2-tuples where the first element is the char and the 2nd is the number of repetitions.

Edit: Note that this is what separates itertools.groupby from the SQL GROUP BY semantics: itertools doesn't (and in general can't) sort the iterator in advance, so groups with the same "key" aren't merged.


Another example:

for key, igroup in itertools.groupby(xrange(12), lambda x: x // 5):
    print key, list(igroup)

results in

0 [0, 1, 2, 3, 4]
1 [5, 6, 7, 8, 9]
2 [10, 11]

Note that igroup is an iterator (a sub-iterator as the documentation calls it).

This is useful for chunking a generator:

def chunker(items, chunk_size):
    '''Group items in chunks of chunk_size'''
    for _key, group in itertools.groupby(enumerate(items), lambda x: x[0] // chunk_size):
        yield (g[1] for g in group)

with open('file.txt') as fobj:
    for chunk in chunker(fobj):

Another example of groupby - when the keys are not sorted. In the following example, items in xx are grouped by values in yy. In this case, one set of zeros is output first, followed by a set of ones, followed again by a set of zeros.

xx = range(10)
for group in itertools.groupby(iter(xx), lambda x: yy[x]):
    print group[0], list(group[1])


0 [0, 1, 2]
1 [3, 4, 5]
0 [6, 7, 8, 9]
  • That's interesting, but wouldn't itertools.islice be better for chunking an iterable? It returns an object that iterates like a generator, but it uses C code. – trojjer Dec 4 '13 at 10:37
  • @trojjer islice would be better IF the groups are consistent sized. – woodm1979 Dec 17 '13 at 17:48


The syntax list(groupby(...)) won't work the way that you intend. It seems to destroy the internal iterator objects, so using

for x in list(groupby(range(10))):

will produce:


Instead, of list(groupby(...)), try [(k, list(g)) for k,g in groupby(...)], or if you use that syntax often,

def groupbylist(*args, **kwargs):
    return [(k, list(g)) for k, g in groupby(*args, **kwargs)]

and get access to the groupby functionality while avoiding those pesky (for small data) iterators all together.

  • 3
    Many of the answers refer to the stumbling block that you must sort before groupby to get expected results. I just encountered this answer, which explains the strange behavior I haven't seen before. I haven't seen before because only now was I trying to list(groupby(range(10)) as @singular says. Before that I'd always used the "recommended" approach of "manually" iterating through the groupby objects rather than letting the list() constructor "automatically" do it. – The Red Pea Sep 11 '14 at 5:13

I would like to give another example where groupby without sort is not working. Adapted from example by James Sulak

from itertools import groupby

things = [("vehicle", "bear"), ("animal", "duck"), ("animal", "cactus"), ("vehicle", "speed boat"), ("vehicle", "school bus")]

    for thing in group:
    print " "

output is

A bear is a vehicle.

A duck is a animal.
A cactus is a animal.

A speed boat is a vehicle.
A school bus is a vehicle.

there are two groups with vehicule, whereas one could expect only one group

  • 4
    You have to sort the data first, using as key the function you are grouping by. This is mentioned in two post above, but is not highlighted. – mbatchkarov Jun 25 '13 at 15:19
  • I was doing a dict comprehension to preserve the sub-iterators by key, until I realised that this was as simple as dict(groupby(iterator, key)). Sweet. – trojjer Dec 4 '13 at 12:00
  • On second thoughts and after experimentation, the dict call wrapped around the groupby will exhaust the group sub-iterators. Damn. – trojjer Dec 4 '13 at 13:57
  • @dgimenes and a cool one indead theoatmeal.com/comics/polar_bear :) – Eloims May 1 '15 at 21:59

@CaptSolo, I tried your example, but it didn't work.

from itertools import groupby 
[(c,len(list(cs))) for c,cs in groupby('Pedro Manoel')]


[('P', 1), ('e', 1), ('d', 1), ('r', 1), ('o', 1), (' ', 1), ('M', 1), ('a', 1), ('n', 1), ('o', 1), ('e', 1), ('l', 1)]

As you can see, there are two o's and two e's, but they got into separate groups. That's when I realized you need to sort the list passed to the groupby function. So, the correct usage would be:

name = list('Pedro Manoel')
[(c,len(list(cs))) for c,cs in groupby(name)]


[(' ', 1), ('M', 1), ('P', 1), ('a', 1), ('d', 1), ('e', 2), ('l', 1), ('n', 1), ('o', 2), ('r', 1)]

Just remembering, if the list is not sorted, the groupby function will not work!


How do I use Python's itertools.groupby()?

You can use groupby to group things to iterate over. You give groupby an iterable, and a optional key function/callable by which to check the items as they come out of the iterable, and it returns an iterator that gives a two-tuple of the result of the key callable and the actual items in another iterable. From the help:


Here's an example of groupby using a coroutine to group by a count, it uses a key callable (in this case, coroutine.send) to just spit out the count for however many iterations and a grouped sub-iterator of elements:

import itertools

def grouper(iterable, n):
    def coroutine(n):
        yield # queue up coroutine
        for i in itertools.count():
            for j in range(n):
                yield i
    groups = coroutine(n)
    next(groups) # queue up coroutine

    for c, objs in itertools.groupby(iterable, groups.send):
        yield c, list(objs)
    # or instead of materializing a list of objs, just:
    # return itertools.groupby(iterable, groups.send)

list(grouper(range(10), 3))


[(0, [0, 1, 2]), (1, [3, 4, 5]), (2, [6, 7, 8]), (3, [9])]

Sorting and groupby

from itertools import groupby

val = [{'name': 'satyajit', 'address': 'btm', 'pin': 560076}, 
       {'name': 'Mukul', 'address': 'Silk board', 'pin': 560078},
       {'name': 'Preetam', 'address': 'btm', 'pin': 560076}]

for pin, list_data in groupby(sorted(val, key=lambda k: k['pin']),lambda x: x['pin']):
...     print pin
...     for rec in list_data:
...             print rec

{'name': 'satyajit', 'pin': 560076, 'address': 'btm'}
{'name': 'Preetam', 'pin': 560076, 'address': 'btm'}
{'name': 'Mukul', 'pin': 560078, 'address': 'Silk board'}

One useful example that I came across may be helpful:

from itertools import groupby

#user input

myinput = input()

#creating empty list to store output

myoutput = []

for k,g in groupby(myinput):



Sample input: 14445221

Sample output: (1,1) (3,4) (1,5) (2,2) (1,1)


You can write own groupby function:

           def groupby(data):
                kv = {}
                for k,v in data:
                    if k not in kv:
           return kv

     Run on ipython:
       In [10]: data = [('a', 1), ('b',2),('a',2)]

        In [11]: groupby(data)
        Out[11]: {'a': [1, 2], 'b': [2]}
  • reinventing wheel is not a great idea, also question is to explain itertools groupby, not writing own – user2678074 May 16 at 8:39
  • 1
    @user2678074 You are right. It's something if you want to write own for a learning point of view. – Sky May 16 at 11:17
  • 1
    Also better use a defaultdict(list) so it's even shorter – Mickey Perlstein Aug 12 at 19:24

protected by codeforester Oct 10 '18 at 18:07


Would you like to answer one of these unanswered questions instead?
