How to remove duplicate objects from an array of objects in JavaScript

In this post we will be covering how to remove duplicate objects from an array of objects in JavaScript with as little technical jargon as possible so you will have everything you need right here without having to look any further!

Removing duplicate objects from an array of objects in JavaScript can be quite tricky in comparison to removing duplicate values such as strings or numbers.

If you do want to find out how to remove duplicates from an array in JavaScript checkout my post about it here.

In this post we will instead be looking specifically at how we can deduplicate or remove duplicate objects from an array of objects in JavaScript.

Before we get stuck in, object comparison is complicated and requires a lot of time and thought for each implementation of it so there is not necessarily a one size fits all approach.

In this post I will go over a potential generic solution, but you should make sure whichever solution you use is well tested. The best solution for you will most likely need to be custom that meets your needs and requirements.

With that said, let’s get started.

How to remove duplicate objects from an array of objects in JavaScript

The best solution to remove duplicate objects from an array of objects in JavaScript is to not allow duplicate objects in an array in the first place, it will save a lot of time, and hassle if you can avoid adding them at the creation.

To remove duplicate objects from an array of objects in JavaScript we are going to need to take a close look at the keys and values of each object property in our array and use that to track identical objects.

This problem comes from how objects will only be equal if they have referential equality, which simply means that even identical objects are different in JavaScript, and only variables that point to the exact same object are classed as equal.

Here is an example of referential equality:

{} === {} // false

const obj = {}
obj === obj // true

Now on top of this, when deduplicating or removing duplicate values from an array it usually refers to simple values such as numbers or strings that can be sorted or easily compared.

Because we are dealing with objects here we need to compare keys, values and properties rather than the objects themselves.

To do this we are going to need to create a helper function to be able to read the keys, values and properties of an object to tell us if we are looking at the same object or not.

In this post we are going to look at shallow comparisons between objects rather than deep comparisons, but the principle will be the same.

One approach we could take to solve this problem is to create a hash for each object, if we have two identical objects then the hash we create would be the same and we can use it for a lookup to see if it already exists within an array.

Let’s take a look at how we can create a simple helper function to do this:

const createHashFromShallow = anObject => {
  const objKeys = Object.keys(anObject).sort().join("")
  const objValues = Object.values(anObject).sort().join("")
  return `${objKeys}${objValues}`
}

It is worth pointing out at this point that this hashing function could be greatly improved upon, but for the purpose of checking for duplicate objects it will work for this example.

It goes without saying that if you choose to implement this method of removing duplicate objects from an array of objects then you should be very careful when implementing your hashing function.

In this function we are using Object.keys, and Object.values to get the keys and values as strings, order them and then combine them into a string to create a very basic hash.

Now we have a hashing function we can combine this with a history object just like when normally removing duplicates from an array in JavaScript.

Let’s combine both of these in this next example:

const createHashFromShallow = anObject => {
  const objKeys = Object.keys(anObject).sort().join("")
  const objValues = Object.values(anObject).sort().join("")
  return `${objKeys}${objValues}`
}

const dedupeArrayOfObjects = someArray => {
  const history = {}
  const newDeduplicatedArray = []

  for (let i = 0; i < someArray.length; i += 1) {
    const hash = createHashFromShallow(someArray[i])

    if (!history?.[hash]) {
      newDeduplicatedArray.push(someArray[i])
      history[hash] = true
    }
  }

  return newDeduplicatedArray
}

const anArrayWithDuplicates = [
  { a: "b", b: "c" },
  { c: "d", d: "e" },
  { a: "b", b: "c" },
  { b: "c", a: "b" },
]
dedupeArrayOfObjects(anArrayWithDuplicates) // [{ a: 'b', b: 'c' }, { c: 'd', d: 'e'}]

As you can see our algorithm correctly removes all duplicated objects, even if the object properties are in different orders.

For the most part this won’t be an issue because most objects will be created in the same order, but the order of an object’s properties is not guaranteed so you can’t rely on it.

That is why we can’t just use something like JSON.stringify to compare objects.

Let’s try this out with another example, but this time with more complex objects.

const today = new Date()
const xmas = new Date("12/25/2020")
const today2 = new Date()

const complexArray = [
  { abc: "def", today, today2, xmas, n: 1234 },
  { abc: "de", today, today2, xmas, n: 1234 },
  { abc: "def", xmas, today2, today, n: 1234 },
  { abc: "def", today2, today, xmas, n: 1234 },
  { abc: "def", today, today2, n: 1234 },
  { abc: "def", today, today2, xmas, n: 1234, extra: "example" },
]

dedupeArrayOfObjects(complexArray) // [{ abc: 'def', today, today2, xmas, n: 1234  }, { abc: 'de', today, today2, xmas, n: 1234  }, { abc: 'def', today, today2, n: 1234  }, { abc: 'def', today, today2, xmas, n: 1234, extra: 'example'  }]

As you can see in the above example even with a complex array with many different keys, values and properties this algorithm using a hash still works.

The reason why it works is because we are sorting each key and value in our hashing function which means it ignores the order of the properties in each object so that we can instead focus on the keys and values instead.

The fallback for this implementation is that if an object has a nested object it will not hash it correctly because when we call sort on an array in JavaScript without providing a comparison callback function then it will convert each item into a string, and if you call toString on an object you get [object Object] instead of the actual string values.

To get around this we could check if each value is an object or not with something like typeof value === "object" && value !== null, and if it is an object then repeat our hashing function on it, and if not then we can continue to convert it into a string. This kind of logic would work well implemented as a recursive function.

Doing these extra checks would be going into a deep comparison.

Here is an example of how a deep comparison function would look when implemented as a semi recursive function:

const createHashFromDeep = anObject => {
  const objKeys = Object.keys(anObject).sort().join("")
  const objValues = Object.values(anObject)
    .sort()
    .map(value => {
      if (typeof value === "object" && value !== null) {
        return createHashFromDeep(value)
      }
      return value
    })
    .join("")

  return `${objKeys}${objValues}`
}

This same rule applies to any other value in the object that does not respond well when converting to a string for the same issue of the object and it is important to keep this in mind because it could cause issues.

This will be dependent on each array and is something you will likely need to check and test thoroughly before implementing any kind of algorithm.

If you have a set of keys/properties that don’t change between objects, and instead only the values change, then it becomes much easier to manage. Instead of checking for the keys, we could remove them and only check the values.

Lastly, if you can perform manual checks on each of these values instead of looping over them, then that will be the best course of action to avoid any potential issues.

Object comparison is a complex topic and one that you should spend time testing thoroughly to ensure that whichever solution you choose to use works well without failures or edge cases.

Summary

There we have how to remove duplicate objects from an array of objects in JavaScript, if you want more like this be sure to check out some of my other posts!