> DISCOVER MORE BLOGS

Data Transformation Mapping – Can it be Automated?

Team Cleo
Although attempts have been made to automate aspects of mapping, it remains largely a manual activity.

Please note: This post originally appeared on Extol.com (EXTOL has been acquired by Cleo).

In our previous post on this subject, we wrote about the neglected problem of data transformation mapping productivity, and its impact on integration project costs, particularly for B2B integration in companies with many customer relationships (and therefore many documents pairs to map).

Although attempts have been made to automate aspects of mapping, it remains largely a manual activity.  Gartner and other industry analysts have published research on automated mapping methods, particularly Dictionary-based mapping, but real-world implementations haven’t put a serious dent in the mapping productivity problem.  Why is that?

Mapping is a large and complex problem space.   It accounts for the single largest part of integration time and effort, encompassing data validation, translation, value derivation, enrichment, aggregation, and routing.  Defining these outcomes involves two kinds of specification, not one:  source-target matching and transformation.  Most attempts to automate mapping have focused intensively on the first part (matching), but very little on the second (data transformations).  But the biggest obstacle to mapping automation is that specifying the right mapping action may require an understanding of business and data contexts and requirements.  And that requires human decision-making.

So to the degree that mapping automation is possible, it must occur in the context of a broader, human-guided mapping process.  Simply defining a source-target dictionary and “lighting the fuse” won’t produce a map that can be used in a production environment.

Integrating automated mapping methods with the human-guided mapping process imposes critical implementation requirements for integration technology providers, including:

  • Unobtrusive integration of automation methods and human mapping decisions in the UI
  • Support for both source-target matching and data transformation aspects of mapping
  • Automation that works with both familiar and unfamiliar document types and combinations
  • Results that can be verified quickly and easily by humans
  • Configurability to suit different circumstances and preferences
  • Ability to “learn” from and adapt past decisions to future mapping situations

Each of these requirements could be subjects for their own blog posts.  The bottom line, however, is that past attempts to automate mapping have ignored or fallen short in all of these areas.

Automating data transformation mapping isn’t easy, but it is possible.

CIC Platform Page
Best-Kept Secrets to B2B Integration Success GUIDE
How to Gain B2B Integration Control

One of the most efficient and effective ways to improve communication in 2023 is through B2B integration. And while there are plenty of B2B integration platforms and solutions on the market, a large majority of them fall short in one key area — giving you control.

WHITE PAPER WHITE PAPER
Your B2B Integration Strategy is Broken

If you can't support end-to-end API & EDI integration or if onboarding partners and customers is painful.... your B2B integration is broken.  Here's what to do about it.

VIDEO VIDEO
Demo Video Library

Instantly access our library of demo videos to see how Cleo Integration Cloud handles all EDI, non-EDI and API integration use cases.

CASE STUDY CASE STUDY
Mohawk Global Logistics

Mohawk Global Logistics cut EDI costs by $60K a year with Cleo. Here's their success story.

RECENT POSTS