Datastores

📄️ Event Stores Overview

An Event Store serves as a comprehensive repository comprising databases and file directories utilized by Syntasa applications to store and manage data generated through various processes. The primary purpose of an Event Store is to facilitate the efficient storage and retrieval of data, enabling seamless operation of Syntasa applications.

📄️ Getting Started with the Datastores

Syntasa writes all the data to specified folders and databases within the environment where the data is processed. Datastores serve as the storage and retrieval points for this information. Syntasa applications utilize three types of data stores: Event Store, ID Store, and Intelligence Hub. Among these, Event Stores are the most commonly used. Below are links to articles that offer detailed insights into each type of data store.

📄️ Creating and Editing an Event Store

This article will cover the following topics:

📄️ Event Stores Datasets

When an event store is used by any app, a database is created with its tables depending on the type of process we have used in the application. The database for development and production are created separately. If you want to have a look at these tables and their data, you can check it within the syntasa application by navigating to DataStores>>DataSets tab.

📄️ Datastores Dependencies

If you're using a single event store across multiple apps and need to identify which apps are utilizing it, we've got you covered! Simply navigate to the 'Dependencies' tab within the Datastore page. Here, you'll find a comprehensive list of all the apps and/or DataLayers that are currently making use of the event store. Keep in mind, if the datastore hasn't been utilized yet, the list of dependencies will remain empty."

📄️ ID Stores

The Syntasa platform includes a Type of Datastore called ID Store, which consists of databases and file directories for Syntasa app data storage. ID stores serve as a useful way to organize your apps on the backend and determine where your data is stored. They also help distinguish between your development and production data. It's important to note that ID stores are only created and become active when data is stored within them. They are required for any app created with the "ID Graph" template, so make sure you have at least one before attempting to create an app.

📄️ Intelligence Hub

An intelligence hub is a NoSQL database storage location (i.e. Big Table, Hbase, DynamoDB). The main purpose of using an intelligence hub is to post Composer model customer attributes into a centralized location.

📄️ External Datasets - An Overview

In the Syntasa application, when any process is executed, the transformed data (output) is written into a dataset that resides as a Hive table within the Syntasa-managed environment. This behavior was consistent across all application processes prior to the 9.0 release. Users did not have the option to directly read from or write data to locations outside the Syntasa environment; all intermediate and final outputs were managed internally.

📄️ Create External Dataset

Creating a reference to an external dataset in Syntasa is a simple and guided process. The Create Dataset screen allows you to define a dataset that points to a Hive table located outside the Syntasa-managed environment, such as cloud object storage. Once created, this external dataset can be used seamlessly within application processes, just like an internal dataset.

📄️ Choose an External Dataset for an App Process

Once an external dataset is created, it can be easily used as the output destination for an application process in Syntasa. This allows the transformed data generated by a process (for example, a Spark processor) to be written directly to an external location, based on the workflow configuration.