In Java 8 how can I filter a collection using the Stream API by checking the distinctness of a property of each object?

For example I have a list of Person object and I want to remove people with the same name,


Will use the default equality check for a Person object, so I need something like,

persons.stream().distinct(p -> p.getName());

Unfortunately the distinct() method has no such overload. Without modifying the equality check inside the Person class is it possible to do this succinctly?

23 Answers 23


Consider distinct to be a stateful filter. Here is a function that returns a predicate that maintains state about what it's seen previously, and that returns whether the given element was seen for the first time:

public static <T> Predicate<T> distinctByKey(Function<? super T, ?> keyExtractor) {
    Set<Object> seen = ConcurrentHashMap.newKeySet();
    return t -> seen.add(keyExtractor.apply(t));

Then you can write:


Note that if the stream is ordered and is run in parallel, this will preserve an arbitrary element from among the duplicates, instead of the first one, as distinct() does.

(This is essentially the same as my answer to this question: Java Lambda Stream Distinct() on arbitrary key?)

  • 26
    I guess for better compatibility the argument should be Function<? super T, ?>, not Function<? super T, Object>. Also it should be noted that for ordered parallel stream this solution does not guarantee which object will be extracted (unlike normal distinct()). Also for sequential streams there's additional overhead on using CHM (which is absent in @nosid solution). Finally this solution violates the contract of filter method which predicate must be stateless as stated in JavaDoc. Nevertheless upvoted. – Tagir Valeev Sep 4 '15 at 9:49
  • 2
    @java_newbie The Predicate instance returned by distinctByKey has no idea of whether it's being used within a parallel stream. It uses CHM in case it is being used in parallel, though this adds overhead in the sequential case as Tagir Valeev noted above. – Stuart Marks Aug 12 '16 at 17:49
  • 4
    @holandaGo It will fail if you save and reuse the Predicate instance returned by distinctByKey. But it works if you call distinctByKey each time, so that it creates a fresh Predicate instance each time. – Stuart Marks Mar 14 '17 at 4:49
  • 2
    @Chinmay no, it shouldn't. If you use .filter(distinctByKey(...)). It will execute the method once and return the predicate. So basically the map is already being re-used if you use it properly within a stream. If you would make the map static, the map would be shared for all usages. So if you have two streams using this distinctByKey(), both would use the same map, which isn't what you want. – g00glen00b Jul 11 '17 at 11:00
  • 3
    This is sooo smart and completely non-obvious. Generally this is a stateful lambda and the underlying CallSite will be linked to the get$Lambda method - that will return a new instance of the Predicate all the time, but those instances will share the same map and function as far as I understand. Very nice! – Eugene Jul 31 '17 at 8:29

An alternative would be to place the persons in a map using the name as a key:

persons.collect(toMap(Person::getName, p -> p, (p, q) -> p)).values();

Note that the Person that is kept, in case of a duplicate name, will be the first encontered.

  • 19
    @skiwi: do you think there is a way to implement distinct() without that overhead? How would any implementation know if it has seen an object before without actually remembering all distinct values it has seen? So the overhead of toMap and distinct is very likely the same. – Holger May 19 '14 at 8:38
  • 1
    @Holger I may have been wrong there as I hadn't thought abou the overhead distinct() itself creates. – skiwi May 19 '14 at 8:50
  • 1
    with less number of object, probably this is best concise and readable answer.. – Mohammad Adnan Oct 2 '15 at 16:18
  • 2
    And obviously it messes up the original order of the list – Philipp Nov 7 '16 at 18:56
  • 7
    @Philipp: could be fixed by changing to persons.collect(toMap(Person::getName, p -> p, (p, q) -> p, LinkedHashMap::new)).values(); – Holger Nov 17 '17 at 8:02

You can wrap the person objects into another class, that only compares the names of the persons. Afterwards, you unwrap the wrapped objects to get a person stream again. The stream operations might look as follows:


The class Wrapper might look as follows:

class Wrapper {
    private final Person person;
    public Wrapper(Person person) {
        this.person = person;
    public Person unwrap() {
        return person;
    public boolean equals(Object other) {
        if (other instanceof Wrapper) {
            return ((Wrapper) other).person.getName().equals(person.getName());
        } else {
            return false;
    public int hashCode() {
        return person.getName().hashCode();
  • 13
    This is called the Schwartzian transform – Stuart Caie May 16 '14 at 18:07
  • 5
    @StuartCaie Not really... there's no memoization, and the point is not performance, but adaptation to the existing API. – Marko Topolnik May 16 '14 at 20:21
  • 6
    com.google.common.base.Equivalence.wrap(S) and com.google.common.base.Equivalence.Wrapper.get() could help too. – bjmi Jan 30 '17 at 12:52
  • You could make the wrapper class generic and parametrized by a key extraction function. – Lii Jul 28 '17 at 2:00
  • The equals method can be simplified to return other instanceof Wrapper && ((Wrapper) other).person.getName().equals(person.getName()); – Holger Nov 17 '17 at 8:03

Another solution, using Set. May not be the ideal solution, but it works

Set<String> set = new HashSet<>(persons.size());
persons.stream().filter(p -> set.add(p.getName())).collect(Collectors.toList());

Or if you can modify the original list, you can use removeIf method

persons.removeIf(p -> !set.add(p.getName()));
  • 1
    This is the best answer if you are not using any third party libraries! – Manoj Shrestha Dec 22 '18 at 1:08
  • 1
    using genious idea that Set.add returns true if this set did not already contain the specified element. +1 – Luvie Jul 31 at 10:27

There's a simpler approach using a TreeSet with a custom comparator.

      () -> new TreeSet<Person>((p1, p2) -> p1.getName().compareTo(p2.getName())) 
  • 4
    I think your answer helps towards the ordering and not towards uniqueness. However it helped me set my thoughts on how to do it. Check here: stackoverflow.com/questions/1019854/… – janagn Mar 14 '15 at 22:01
  • Keep in mind you will be paying the price for sorting the elements here and we do not need sorting in order to find duplicates or even remove duplicates. – pisaruk Jul 4 '16 at 20:00
  • 11
    Comparator.comparing(Person::getName) – Jean-François Savard Jun 13 '17 at 20:05

We can also use RxJava (very powerful reactive extension library)



Observable.from(persons).distinct(p -> p.getName())
  • Rx is awesome, but this is a poor answer. Observable is push-based whereas Stream is pull-based. stackoverflow.com/questions/30216979/… – sdgfsdh Jun 1 '17 at 16:25
  • 4
    the question ask for a java8 solution not necessarily using stream. My answer show that java8 stream api is less powefull than rx api – frhack Jun 2 '17 at 0:41
  • 1
    Using reactor, it will be Flux.fromIterable(persons).distinct(p -> p.getName()) – Ritesh Aug 16 '17 at 14:53
  • The question literally says "using the Stream API", not "not necessarily using stream". That said, this is a great solution to the XY problem of filtering the stream to distinct values. – M. Justin May 2 '18 at 16:43

You can use the distinct(HashingStrategy) method in Eclipse Collections.

List<Person> persons = ...;
MutableList<Person> distinct =
    ListIterate.distinct(persons, HashingStrategies.fromFunction(Person::getName));

If you can refactor persons to implement an Eclipse Collections interface, you can call the method directly on the list.

MutableList<Person> persons = ...;
MutableList<Person> distinct =

HashingStrategy is simply a strategy interface that allows you to define custom implementations of equals and hashcode.

public interface HashingStrategy<E>
    int computeHashCode(E object);
    boolean equals(E object1, E object2);

Note: I am a committer for Eclipse Collections.


You can use groupingBy collector:

persons.collect(Collectors.groupingBy(p -> p.getName())).values().forEach(t -> System.out.println(t.get(0).getId()));

If you want to have another stream you can use this:

persons.collect(Collectors.groupingBy(p -> p.getName())).values().stream().map(l -> (l.get(0)));

You can use StreamEx library:

  • Unfortunately, that method of the otherwise awesome StreamEx library is poorly designed - it compares object equality instead of using equals. This may work for Strings thanks to string interning, but it also may not. – Torque Jun 24 at 10:40

I recommend using Vavr, if you can. With this library you can do the following:

                       .toJavaSet() // or any another Java 8 Collection

Extending Stuart Marks's answer, this can be done in a shorter way and without a concurrent map (if you don't need parallel streams):

    final Set<Object> seen = new HashSet<>();
    return t -> seen.add(keyExtractor.apply(t));

Then call:

persons.stream().filter(distinctByKey(p -> p.getName());
  • 2
    This one doesn't take into consideration that the stream might be parallel. – brunnsbe Dec 27 '16 at 12:08
  • Thanks for the comment, I've updated my answer. If you don't need a parallel stream, not using concurrent maps gives you much better performance. – Wojciech Górski Dec 27 '16 at 22:39
  • Your code would probably work for parallel collections if you created a Collections.synchronizedSet(new HashSet<>()) instead. But it would probably be slower than with a ConcurrentHashMap. – Lii Jul 28 '17 at 2:13

I made a generic version:

private <T, R> Collector<T, ?, Stream<T>> distinctByKey(Function<T, R> keyExtractor) {
    return Collectors.collectingAndThen(
                    t -> t,
                    (t1, t2) -> t1
            (Map<R, T> map) -> map.values().stream()

An exemple:

Stream.of(new Person("Jean"), 
          new Person("Jean"),
          new Person("Paul")
    .collect(distinctByKey(Person::getName)) // return a stream of Person with 2 elements, jean and Paul
Set<YourPropertyType> set = new HashSet<>();
        .filter(it -> set.add(it.getYourProperty()))
        .forEach(it -> ...);

Similar approach which Saeed Zarinfam used but more Java 8 style:)

 .map(plans -> plans.stream().findFirst().get())

Another library that supports this is jOOλ, and its Seq.distinct(Function<T,U>) method:


Under the hood, it does practically the same thing as the accepted answer, though.


My approach to this is to group all the objects with same property together, then cut short the groups to size of 1 and then finally collect them as a List.

  List<YourPersonClass> listWithDistinctPersons =   persons.stream()
            //operators to remove duplicates based on person name
            .collect(Collectors.groupingBy(p -> p.getName()))
            //cut short the groups to size of 1
            .flatMap(group -> group.stream().limit(1))
            //collect distinct users as list

Distinct objects list can be found using:

 List distinctPersons = persons.stream()
                            Collectors.toCollection(() -> new TreeSet<>(Comparator.comparing(Person:: getName))),

The easiest way to implement this is to jump on the sort feature as it already provides an optional Comparator which can be created using an element’s property. Then you have to filter duplicates out which can be done using a statefull Predicate which uses the fact that for a sorted stream all equal elements are adjacent:

Comparator<Person> c=Comparator.comparing(Person::getName);
stream.sorted(c).filter(new Predicate<Person>() {
    Person previous;
    public boolean test(Person p) {
      if(previous!=null && c.compare(previous, p)==0)
        return false;
      return true;
})./* more stream operations here */;

Of course, a statefull Predicate is not thread-safe, however if that’s your need you can move this logic into a Collector and let the stream take care of the thread-safety when using your Collector. This depends on what you want to do with the stream of distinct elements which you didn’t tell us in your question.


Building on @josketres's answer, I created a generic utility method:

You could make this more Java 8-friendly by creating a Collector.

public static <T> Set<T> removeDuplicates(Collection<T> input, Comparator<T> comparer) {
    return input.stream()
            .collect(toCollection(() -> new TreeSet<>(comparer)));

public void removeDuplicatesWithDuplicates() {
    ArrayList<C> input = new ArrayList<>();
    Collections.addAll(input, new C(7), new C(42), new C(42));
    Collection<C> result = removeDuplicates(input, (c1, c2) -> Integer.compare(c1.value, c2.value));
    assertEquals(2, result.size());
    assertTrue(result.stream().anyMatch(c -> c.value == 7));
    assertTrue(result.stream().anyMatch(c -> c.value == 42));

public void removeDuplicatesWithoutDuplicates() {
    Collections.addAll(input, new C(1), new C(2), new C(3));
    Collection<C> result = removeDuplicates(input, (t1, t2) -> Integer.compare(t1.value, t2.value));
    assertEquals(3, result.size());
    assertTrue(result.stream().anyMatch(c -> c.value == 1));
    assertTrue(result.stream().anyMatch(c -> c.value == 2));
    assertTrue(result.stream().anyMatch(c -> c.value == 3));

private class C {
    public final int value;

    private C(int value) {
        this.value = value;

Maybe will be useful for somebody. I had a little bit another requirement. Having list of objects A from 3rd party remove all which have same A.b field for same A.id (multiple A object with same A.id in list). Stream partition answer by Tagir Valeev inspired me to use custom Collector which returns Map<A.id, List<A>>. Simple flatMap will do the rest.

 public static <T, K, K2> Collector<T, ?, Map<K, List<T>>> groupingDistinctBy(Function<T, K> keyFunction, Function<T, K2> distinctFunction) {
    return groupingBy(keyFunction, Collector.of((Supplier<Map<K2, T>>) HashMap::new,
            (map, error) -> map.putIfAbsent(distinctFunction.apply(error), error),
            (left, right) -> {
                return left;
            }, map -> new ArrayList<>(map.values()),
            Collector.Characteristics.UNORDERED)); }

In my case I needed to control what was the previous element. I then created a stateful Predicate where I controled if the previous element was different from the current element, in that case I kept it.

public List<Log> fetchLogById(Long id) {
    return this.findLogById(id).stream()
        .filter(new LogPredicate())

public class LogPredicate implements Predicate<Log> {

    private Log previous;

    public boolean test(Log atual) {
        boolean isDifferent = previouws == null || verifyIfDifferentLog(current, previous);

        if (isDifferent) {
            previous = current;
        return isDifferent;

    private boolean verifyIfDifferentLog(Log current, Log previous) {
        return !current.getId().equals(previous.getId());


The Most simple code you can write:

    persons.stream().map(x-> x.getName()).distinct().collect(Collectors.toList());
  • 12
    That'll get a distinct list of names though, not Persons by name – RichK Oct 8 '17 at 8:13

If you want to List of Persons following would be the simple way


Additionally, if you want to find distinct or unique list of names, not Person , you can do using following two method as well.

Method 1: using distinct


Method 2: using HashSet

Set<E> set = new HashSet<>();
  • 2
    This produces a list of names, not Persons. – Hulk Dec 6 '18 at 11:21
  • 1
    This is exactly what I was looking for. I needed a single line method to eliminate duplicates while transforming a collection to one other. Thanks. – Raj Dec 18 '18 at 5:29

protected by cassiomolin Sep 18 at 12:35


Would you like to answer one of these unanswered questions instead?
