Modular Orchestration: Architecture Overview
Netel ETL Engine addresses the challenges of monolithic and difficult-to-maintain ETL processes through the concept of ChildJobs (subtasks). Instead of a single large script, complex business logic and data flows are decomposed into smaller, isolated, and reusable components.
This hierarchical structure enables precise dependency management, maximum hardware utilization, and advanced execution control, with particular emphasis on parallel and conditional processing.
Parallel Execution
The parallel execution model is designed for maximum speed and significant reduction in processing time during critical phases such as daily data loading and analytical report preparation.
- Maximum Throughput: The system simultaneously executes multiple independent ChildJobs (e.g., loading transaction tables, master data, and customer profiles from different sources).
- Asynchronous Processing: Tasks run independently, eliminating bottlenecks commonly found in traditional sequential systems.
Conditional Execution (If-Then-Else Control)
Conditional execution introduces intelligence and flexibility into the data pipeline, enabling real-time adaptation based on input parameters or system conditions.
- Dynamic Workflows: Netel ETL Engine manages branching logic entirely through JSON-as-Code. Workflow execution depends dynamically on input parameters evaluated in real time. Simple if-then-else rules allow the system to activate ChildJob A when conditions are met or ChildJob B when they are not.
- Intelligent Error Handling and Chain Control: Conditional logic and strict hierarchy ensure complete data consistency and prevent processing of incomplete datasets. If any critical ChildJob fails, execution stops immediately and all remaining tasks are automatically canceled. The error is isolated in log files, and an email notification is sent to administrators, ensuring that production environments are never contaminated with partially loaded data.
- Reduced Redundancy: Resource-intensive analytical processes are skipped when no new records are detected in source systems, saving resources and improving infrastructure stability.
Additional Execution Models Within the ChildJobs Hierarchy
In addition to parallel and conditional execution, Netel ETL Engine supports five specialized execution models:
- Sequential Execution: A strictly defined sequence where each task begins only after the successful completion of the previous task.
- Data Transfer Task: Direct transfer, mapping, and transformation of data between heterogeneous sources and destinations using the ODBC standard.
- DTI (Delete-Then-Insert) Task: An atomic operation that removes existing records and inserts new data, ensuring reporting consistency without duplication.
- Stored Procedure Task: Execution of native, optimized database procedures directly on the target database.
- Email Task: Automated notification system that instantly distributes detailed status information and logs based on ChildJob outcomes.
