Informatica Data Engineering Integration for Developers

To optimize the performance of a Data Engineering system, it’s essential to implement monitoring and troubleshooting techniques. Regular monitoring allows for identifying bottlenecks, resource usage patterns, and potential issues before they escalate. By analyzing these insights, engineers can make informed decisions about optimizing resource allocation, improving data processing pipelines, and reducing latency.

Learn to accelerate Data Engineering Integration through mass ingestion, incremental loads, transformations, processing of complex files, creating dynamic mappings, and integrating data science using Python.

Course Key Learnings:

• Mass ingest data to Hive and HDFS
• Perform incremental loads in Mass Ingestion
• Perform initial and incremental loads
• Integrate with relational databases using SQOOP
• Perform transformations across various engines
• Execute a mapping using JDBC in Spark mode
• Perform stateful computing and windowing
• Process complex files
• Parse hierarchical data on Spark engine
• Run profiles and choose sampling options on Spark engine
• Execute Dynamic Mappings
• Create Audits on Mappings
• Monitor logs using REST Operations Hub
• Monitor logs using Log Aggregation and troubleshoot
• Run mappings in Databricks environment
• Create mappings to access Delta Lake tables
• Tune performances of Spark and Databricks jobs

Course Content:

Module 1: Informatica Data Engineering Management Overview

• Data Engineering concepts
• Data Engineering Management features
• Benefits of Data Engineering Management
• Data Engineering Management architecture
• Data Engineering Management developer tasks
• Data Engineering Integration 10.5 new features

Module 2: Ingestion and Extraction

• Integrating Data Engineering Integration with Hadoop cluster
• Application Services of Data Engineering Integration 10.4.0
• Hadoop file systems
• Ingest data to HDFS and Hive using SQOOP
• Mass Ingestion to HDFS and Hive – Initial load
• Mass Ingestion to HDFS and Hive – Incremental load

Module 3: Native and Hadoop Engine Strategy

• DEI engine strategy
• Hive Engine architecture
• MapReduce
• Tez
• Spark architecture
• Blaze architecture

Module 6: Hierarchical Data Processing 

• Hierarchical Data Processing
• Flatten Hierarchical Data
• Dynamic Flattening with Schema Changes
• Hierarchical Data Processing with Schema Changes
• Complex Configuration, Operators and Functions
• Dynamic Ports
• Dynamic Input Rules
• Lab: Flattening a complex port in a Mapping
• Lab: Building dynamic mappings using dynamic ports
• Lab: Building dynamic mappings using input rules
• Lab: Performing Dynamic Flattening of complex ports
• Lab: Parsing Hierarchical Data on the Spark Engine