Process Modes
Process Modes - Overview
Data processing pipelines in Syntasa offer multiple process modes to efficiently manage data ingestion and updates. These modes are designed to handle different use cases, ensuring that data is either added, updated, or entirely replaced based on the required workflow. Choosing the right process…
Drop & Replace - Process Mode
The Drop & Replace mode is designed to completely overwrite an existing dataset with new incoming data during a job execution. This mode is useful when you want to ensure that the latest dataset reflects only the new data and does not retain any previous records.
Replace Date Range - Process Modes
In data science workflows, especially in time-partitioned datasets, managing data updates with precision is critical. Syntasa’s Replace Data Range process mode provides a reliable and controlled method to update output tables based on a specified execution date range.
Add New Only- Process Modes
The Add New Only process mode in Syntasa ensures that only new data partitions—those that do not already exist in the output table—are processed and added. Unlike other process modes, it does not overwrite or modify existing date partitions, making it an ideal choice when you only need to add new…
Add New & Replace Modified - Process Mode
The Add New & Replace Modified process mode in Syntasa is designed to efficiently handle data updates by incorporating new date partitions while ensuring that existing date partitions are updated if they have changed. This mode ensures that your dataset remains up to date by checking for both newly…
Process Mode — “Code Managed”
Within Syntasa, Process Modes define how the platform manages the lifecycle of output data generated by a process. In standard execution modes, Syntasa automatically handles operational behaviors such as partition cleanup, date-range filtering, incremental processing, table writes, and…
Handling Partitioned Data with “Code Managed” Process Mode
In a standard Syntasa workflow, the platform’s State Service automatically manages the lifecycle of partitioned tables. When you select a mode like Replace Date Range , the platform identifies which partitions exist in the target, drops them for the specified window, and writes new data. However,…
Managing Non-Partitioned Data with “Code Managed” Process Mode
For non-partitioned tables, Syntasa’s standard process modes typically operate on an “all-or-nothing” basis. For example, the Drop & Replace mode deletes the entire table and recreates it with the current session’s data.
Contextual Guidance - Process Modes
As data orchestration workflows become more advanced, selecting the correct execution strategy is essential for maintaining data integrity, optimizing performance, and reducing unnecessary processing costs. To help users make more informed decisions, Syntasa 9.1 introduces Contextual Help for…