Python Sample Codes for Spark Processor
How to Copy a CSV File from Google Cloud Storage to Amazon S3 Using Python
As organizations move toward hybrid and multi-cloud architectures, it’s increasingly common to work with data spread across multiple cloud providers. Two popular services in this domain are Google Cloud Storage (GCS) and Amazon S3. Sometimes, you may need to move data between them — for instance,…
Reading a CSV file from GCS bucket on AWS
In this article, we will walk through the process of reading a CSV file stored in a Google Cloud Storage (GCS) bucket from an application running in an AWS environment using PySpark. This is particularly useful in multi-cloud architectures where data may reside in GCS while processing is done on…
Creating a Simple Propensity Model in Syntasa App
Running code in a Syntasa app differs from working in an interactive environment, such as Syntasa Notebooks or JupyterLab. In the Syntasa app, your code is written similarly to how you would in a script or an IDE, such as VS Code — it is executed all at once, not cell by cell.
Reading CSV files from S3 bucket (Instance profile Disabled)
In the previous article, we explored how to read a CSV file from an S3 bucket located within the same AWS environment, using an instance profile for authentication. In some cases, the S3 bucket you want to access is not hosted within the same AWS environment as your Syntasa application. This…
Reading CSV File from S3 Bucket (Instance Profile Enabled)
In Syntasa, Spark processors can be configured to access S3 buckets without hardcoding credentials by leveraging instance profiles. Before running this Spark code within the Syntasa application, it’s common to first test and validate the logic using Syntasa/JupyterLab notebook . If you’d like to…
Reading Data from Snowflake with Notebooks
Snowflake is a cloud-native data platform that separates compute from storage, letting you scale each independently. This guide shows you how to pull data out of Snowflake into your notebook using Python, and outlines two options, then demonstrates how to join web data with your offline (Snowflake)…
Writing Data to Snowflake with Spark Processor
This guide explains how to extract data using Python Spark (PySpark) SQL, transform the data, and write it to a Snowflake table.
Reading data from Snowflake with Spark Processor
This guide explains how to read data using Python Spark (PySpark) SQL and write Hive table.