Remove Duplicates

Use the Remove Duplicates node to identify and delete items that are identical across all fields or a subset of fields. This is helpful in situations where you can end up with duplicate data, such as a user creating multiple accounts, or a customer submitting the same order multiple times. When working with large datasets it becomes more difficult to spot and remove these items.

Node parameters

• Compare : specify which fields of the input data Mosaic Workflows should compare to check if they're the same. The following options are available:
  • All Fields : compares all fields of the input data.
  • All Fields Except : enter which input data fields Mosaic Workflows should exclude from the comparison. You can provide multiple values separated by commas.
  • Selected Fields : enter which input data fields Mosaic Workflows should include in the comparison. You can provide multiple values separated by commas.
• If you choose All Fields Except or Selected Fields , Mosaic Workflows displays Options > Add Field . Use this to add more optional settings, including:
  • Disable Dot Notation : when disabled, you can't reference child fields (in the format parent.child ).
  • Remove Other Fields : keep the fields that you're comparing and remove the others.

Related resources

Learn more about data structure and data flow in Mosaic Workflows workflows.