Syntasa Notebook Utilities

Notebook Utilities Reference

Syntasa Notebook Utilities Reference — Python & Scala

synutils is a pre-initialized utility object available in every Syntasa Notebook. It provides access to credential management, cloud storage, infrastructure configuration, Spark initialization, package installation, and more.

Python

# synutils is already available — no import needed
synutils.credentials.list()
synutils.fs.upload("/tmp/data.csv", "my-bucket/data.csv")
synutils.lib.installPyPI("pandas")

Scala

// synutils is already available — no import needed
synutils.credentials.list()
synutils.fs.upload("/tmp/data.csv", "my-bucket/data.csv")
infrastructure.getProviderType()

Available Modules

Module	Description	Language
`synutils.credentials`	Securely store, read, share, and manage secrets via Vault	Both
`synutils.fs`	Cloud storage operations (S3, GCS, Azure Blob)	Both
`synutils.infrastructure` / `infrastructure`	Platform infrastructure configuration	Both
`synutils.lib`	Install pip/conda packages at runtime	Python
`synutils.kernel`	Restart the notebook kernel	Python
`synutils.ray`	Create and manage Ray clusters	Python
`synutils.geo_lib`	Register GeoMesa geospatial UDFs in Spark	Both
`synutils.auth` / `synutils.authMetadata`	Access authentication tokens	Both
`synutils.configuration`	Platform configuration key-value store	Python

Scala aliases: In Scala notebooks, infrastructure and runtime are available as top-level aliases. All members of synutils (like fs, credentials) are also imported directly.

Details

Credential Store

Securely store, retrieve, update, share, and manage credentials (API keys, passwords, tokens) using HashiCorp Vault. Credential values are automatically redacted in notebook output to prevent accidental exposure.

create Both

synutils.credentials.create(credential_store_name, data)

Create a new credential store.

Python

synutils.credentials.create("my_api_keys", {
    "api_key": "abc123",
    "api_secret": "xyz789"
})

Scala

synutils.credentials.create("my_api_keys", Map(
    "api_key" -> "abc123",
    "api_secret" -> "xyz789"
))

Parameter	Type	Description
`credential_store_name`	`str` / `String`	Name for the credential store (must be lowercase)
`data`	`dict` / `Map[String, String]`	Key-value pairs to store

read Both

synutils.credentials.read(name) → SecretDict / SecretMap
synutils.credentials.read(name, key=...) → SecretString

Read credentials. Returns all keys as a redacted dictionary, or a single redacted value.

Python

# Read all keys — returns SecretDict
creds = synutils.credentials.read("my_api_keys")
# {'api_key': '[REDACTED]', 'api_secret': '[REDACTED]'}
# Read a specific key — returns SecretString
api_key = synutils.credentials.read("my_api_keys", key="api_key")
# [REDACTED]

Scala

// Read all keys — returns SecretMap
val creds: SecretMap = synutils.credentials.read("my_api_keys")
// Map(api_key -> [REDACTED], api_secret -> [REDACTED])

// Read a specific key — returns SecretString
val apiKey: SecretString = synutils.credentials.read("my_api_keys", "api_key")
// [REDACTED]

Using Credentials with SDKs

Python

Use .unseal() to get the real value when passing to SDKs that internally call str():

creds = synutils.credentials.read("aws_creds")
raw = creds.unseal()  # Returns plain dict with real values

client = boto3.client('s3',
    aws_access_key_id=raw['access_key'],
    aws_secret_access_key=raw['secret_key']
)

Scala

SecretString has an implicit conversion to String, so it works directly with JDBC, Spark, and most SDKs:

val password: SecretString = synutils.credentials.read("db_creds", "password")

// Works directly — implicit conversion
DriverManager.getConnection(url, user, password)

// For string interpolation, use type ascription
val jdbcUrl = s"jdbc:postgresql://host/${password: String}"

update Both

synutils.credentials.update(name, updated_kv=...) # update multiple
synutils.credentials.update(name, key=..., value=...) # update single

Python

# Update a single key
synutils.credentials.update("my_api_keys", key="api_key", value="new_key_123")
# Update multiple keys
synutils.credentials.update("my_api_keys", updated_kv={
    "api_key": "new_key_123",
    "api_secret": "new_secret_456"
})

Scala

// Update a single key
synutils.credentials.update("my_api_keys", key = "api_key", value = "new_key_123")

// Update multiple keys
synutils.credentials.update("my_api_keys", updatedData = Map(
    "api_key" -> "new_key_123"
))

delete Both

synutils.credentials.delete(name)

synutils.credentials.delete("my_api_keys")

list Both

synutils.credentials.list()

Lists all credential stores for the current user. Python prints a formatted PrettyTable; Scala prints names to stdout.

synutils.credentials.list()
# +----------------+
# |  Credentials   |
# +----------------+
# | my_api_keys    |
# | db_passwords   |
# +----------------+

share Both

synutils.credentials.share(username, permissions, credential_name)

Python

synutils.credentials.share(
    username="colleague@company.com",
    permissions=["read"],
    credential_name="my_api_keys"
)

Scala

synutils.credentials.share(
    username = "colleague@company.com",
    permissions = List("read"),
    credentialStoreName = "my_api_keys"
)

Parameter	Type	Description
`username`	`str` / `String`	Username to share with
`permissions`	`list` / `List[String]`	Permissions: `"read"`, `"create"`, `"update"`, `"delete"`
`credential_name`	`str` / `String`	Credential store name (must be lowercase)

revoke Both

synutils.credentials.revoke(username, permissions, credential_name)

Revoke a user's access. Same parameters as share.

File System

Cloud storage operations for uploading, downloading, listing, and managing files. The implementation is automatically selected based on your cloud provider (AWS S3, Google Cloud Storage, or Azure Blob Storage).

upload Both

synutils.fs.upload(src_path, dest_path)

synutils.fs.upload("/tmp/report.csv", "my-bucket/reports/report.csv")

upload_folder / uploadFolder Both

synutils.fs.upload_folder(src, dest) # Python
synutils.fs.uploadFolder(src, dest) // Scala

synutils.fs.upload_folder("/tmp/output/", "my-bucket/output/")  # Python
synutils.fs.uploadFolder("/tmp/output/", "my-bucket/output/")   // Scala

download Both

synutils.fs.download(src_path, dest_path)

synutils.fs.download("my-bucket/data/input.csv", "/tmp/input.csv")

download_folder / downloadFolder Both

synutils.fs.download_folder(src, dest) # Python
synutils.fs.downloadFolder(src, dest) // Scala

list Both

synutils.fs.list(path) → list / List[String]

files = synutils.fs.list("my-bucket/data/")

list_file_paths Python

synutils.fs.list_file_paths(path) → list

List full file paths in a directory. Available on S3 and Azure.

is_exists / isExists Both

synutils.fs.is_exists(path) → bool # Python
synutils.fs.isExists(path) → Boolean // Scala

if synutils.fs.is_exists("my-bucket/data/input.csv"):
    print("File found!")

delete Both

synutils.fs.delete(path)

synutils.fs.delete("my-bucket/old-data/temp.csv")

move Python

synutils.fs.move(src_path, dest_path, exclude_folders=None, rewrite_subfolders=None)

Move or copy files within cloud storage.

synutils.fs.move("my-bucket/temp/", "my-bucket/final/")
# With exclusions
synutils.fs.move("my-bucket/src/", "my-bucket/dest/",
    exclude_folders=["logs", "temp"])

Note: Implemented for S3 and GCS. Not available on Azure or in Scala.

stream Python

synutils.fs.stream(path)

Stream file content from cloud storage. Returns a file-like object. Available on S3 and GCS.

upload_stream Python

synutils.fs.upload_stream(file, dest_path)

Upload from a file-like object to cloud storage.

Note: Currently implemented for S3 only.

read Scala

synutils.fs.read(path): Array[Byte]

Read file contents as a byte array.

val content: Array[Byte] = synutils.fs.read("my-bucket/config.json")
val text = new String(content, "UTF-8")

Scala-Only Properties

synutils.fs.PREFIX           // "s3://", "gs://", etc.
synutils.fs.getBaseDir()     // User's base workspace directory
synutils.fs.getLocalTempDir() // Local temporary directory

Infrastructure

Access platform infrastructure configuration: cloud provider type, region, storage, metastore, network, and security settings.

Python

synutils.infrastructure.get_provider_type_from_metadata()
# Returns: "GOOGLE", "AMAZON", or "AZURE"

synutils.infrastructure.get_config_from_metadata()     # Region, project ID, etc.
synutils.infrastructure.get_storage_from_metadata()    # Bucket names, paths
synutils.infrastructure.get_metastore_from_metadata()  # Hive, Glue, BigQuery settings
synutils.infrastructure.get_security_from_metadata()   # Cloud credentials, SSH
synutils.infrastructure.get_network_from_metadata()    # Network config
synutils.infrastructure.get_global_init_script_from_metadata() # Init script or None

Scala

infrastructure.getProviderType()
// Returns: "AWS", "GOOGLE", or "AZURE"

infrastructure.getConfig()     // Map[String, AnyRef]
infrastructure.getStorage()    // Map[String, AnyRef]
infrastructure.getMetastore()  // Map[String, AnyRef]
infrastructure.getSecurity()   // Map[String, AnyRef]
infrastructure.getNetwork()    // Map[String, AnyRef]

Library Management Python

Install Python packages at runtime. Packages are installed locally and automatically distributed to Spark executor pods on Kubernetes.

installPyPI

synutils.lib.installPyPI(packages)

synutils.lib.installPyPI("pandas")
synutils.lib.installPyPI("pandas numpy scikit-learn")  # Space-separated
synutils.lib.installPyPI("requests>=2.28")              # Version constraint

Tip: Installed packages are tracked and automatically installed on Spark executor pods when the session refreshes.

installCondaPackage

synutils.lib.installCondaPackage(packages)

synutils.lib.installCondaPackage("scipy")
synutils.lib.installCondaPackage("scipy numpy")  # Space-separated

Installs from the conda-forge channel.

Kernel Utilities Python

restart_kernel

synutils.kernel.restart_kernel(execute_cell_below=False)

synutils.kernel.restart_kernel()                        # Simple restart
synutils.kernel.restart_kernel(execute_cell_below=True) # Restart + run cells below

Parameter	Type	Default	Description
`execute_cell_below`	`bool`	`False`	Execute all cells below after restart

Spark Initialization

Spark is automatically initialized when your notebook starts. The spark variable (SparkSession) is available in your notebook namespace once initialization completes.

Python — Manual Control

synutils.start_spark_initialization()   # Re-trigger initialization
synutils.cancel_spark_initialization()  # Cancel initialization

Python — Advanced Configuration

from syn_utils.syn_notebook.utils.spark.spark_helper_utils import SparkHelperUtils
# Auto-detect resource manager
spark = SparkHelperUtils.initialize(driver_memory="4g", executor_memory="8g")
# Kubernetes-specific
spark = SparkHelperUtils.initialize_with_kubernetes(driver_memory="4g", executor_memory="8g")
# YARN-specific (requires cluster master IP)
spark = SparkHelperUtils.initialize_with_yarn(syntasa_cluster_master_ip="10.0.0.1")
# Local mode
spark = SparkHelperUtils.initialize_local_spark(driver_memory="4g")
# Ray-based Spark
spark = SparkHelperUtils.initialize_ray_spark(
    driver_memory="4g", driver_cores="2",
    executor_memory="8g", executor_cores="4")

Scala — SparkInitializer

import com.syntasa.kernel.utils.spark.SparkInitializer

SparkInitializer(
    name = "MyNotebook",
    resourceManager = "KUBERNETES",  // or "YARN", "LOCAL"
    driverMemory = Some("4g"),
    executorMemory = Some("8g"),
    executorCores = Some("4"),
    userConfiguration = Map("spark.sql.shuffle.partitions" -> "200")
)

// `spark` is now available globally
spark.sql("SELECT * FROM my_table")

SparkInitializer Parameters (Scala)

Parameter	Type	Default	Description
`name`	`String`	(required)	Spark session name
`resourceManager`	`String`	(required)	`"KUBERNETES"`, `"YARN"`, or `"LOCAL"`
`metastoreUrl`	`Option[String]`	`None`	Hive metastore JDBC URL
`metastoreUsername`	`Option[String]`	`None`	Metastore username
`metastorePassword`	`Option[String]`	`None`	Metastore password
`driverMemory`	`Option[String]`	`None`	e.g., `"4g"`
`driverCore`	`Option[String]`	`None`	Driver CPU cores
`executorMemory`	`Option[String]`	`None`	e.g., `"8g"`
`executorCores`	`Option[String]`	`None`	Executor CPU cores
`clusterAddress`	`Option[String]`	`None`	Cluster master IP (YARN)
`userConfiguration`	`Map[String, String]`	`Map()`	Custom Spark properties
`useDedicatedNodes`	`Boolean`	`false`	Dedicated compute nodes (K8s)

Metastore: If all three metastore parameters are provided, JDBC metastore is used. Otherwise, AWS Glue metastore is used.

Ray Cluster Management Python

Create and manage Ray clusters for distributed computing.

create_cluster

synutils.ray.create_cluster(template_name)

cluster = synutils.ray.create_cluster("my-ray-template")

ray_init

synutils.ray.ray_init(kernel_id)

Initialize connection to a Ray cluster.

setup_spark

synutils.ray.setup_spark(debug=False, additional_spark_config={})

spark = synutils.ray.setup_spark()
spark = synutils.ray.setup_spark(debug=True, additional_spark_config={"spark.executor.memory": "8g"})

Returns: SparkSession

delete

synutils.ray.delete(provider_type, workflow_id, security, config)

Delete a Ray cluster and its resources.

GeoMesa (Geospatial) Both

Register GeoMesa/Sedona geospatial UDFs in your Spark session for spatial queries. Functions are registered with the sedona_ prefix.

Python

synutils.geo_lib.register(spark)
df = spark.sql("SELECT sedona_ST_Point(lon, lat) as geometry FROM locations")

Scala

GeoMesa is automatically registered during SparkInitializer.apply().

spark.sql("SELECT sedona_ST_Point(lon, lat) as geometry FROM locations")

Run Notebook Python

synutils.run_notebook(path)

Execute another notebook in the current kernel context. The notebook is downloaded from cloud storage and run locally. Variables and functions defined in the target notebook become available in the current session.

synutils.run_notebook("my_folder/shared_functions.ipynb")

Parameter	Type	Description
`path`	`str`	Path relative to the user workspace (`syn-workspace/users/`)

Authentication Both

Access and manage authentication tokens. Primarily used internally, but available for making authenticated API calls.

Python

# Token access
token = synutils.auth.get_access_token()
refresh = synutils.auth.get_refresh_token()
auth_type = synutils.auth.get_auth_type()
# Token validation & refresh
valid_token = synutils.auth_client.validate_token_from_metadata()
result = synutils.auth_client.validate_token(token)
# Login
response = synutils.auth_client.login("user@company.com", "password", "LOCAL")
new_tokens = synutils.auth_client.refresh_token("LOCAL", refresh_token)
# User info
user_info = synutils.auth_client.get_user_info_by_name("user@company.com")

Scala

val token: String = synutils.authMetadata.validateTokenFromMetadata()

Configuration Python

synutils.configuration.get("key_name")
synutils.configuration.set("key_name", "value")

provider = synutils.configuration.get_cloud_provider_type()
# Returns: "GOOGLE", "AMAZON", or "AZURE"
# Static method — get environment variable
from syn_utils.syn_auth.config import Configuration
env_val = Configuration.get_env("MY_ENV_VAR", default="fallback")

Quick Reference

Python

# --- Credential Store ---
synutils.credentials.create("name", {"key": "val"})
synutils.credentials.read("name")                          # → SecretDict
synutils.credentials.read("name", key="key")               # → SecretString
synutils.credentials.update("name", key="k", value="v")
synutils.credentials.update("name", updated_kv={"k": "v"})
synutils.credentials.delete("name")
synutils.credentials.list()
synutils.credentials.share("user", ["read"], "name")
synutils.credentials.revoke("user", ["read"], "name")
# --- File System ---
synutils.fs.upload("/local/file", "remote/file")
synutils.fs.download("remote/file", "/local/file")
synutils.fs.upload_folder("/local/dir/", "remote/dir/")
synutils.fs.download_folder("remote/dir/", "/local/dir/")
synutils.fs.list("remote/path/")
synutils.fs.list_file_paths("remote/path/")
synutils.fs.is_exists("remote/file")
synutils.fs.delete("remote/file")
synutils.fs.move("src/", "dest/")
synutils.fs.stream("remote/file")
synutils.fs.upload_stream(file_obj, "remote/file")
# --- Infrastructure ---
synutils.infrastructure.get_provider_type_from_metadata()
synutils.infrastructure.get_config_from_metadata()
synutils.infrastructure.get_storage_from_metadata()
synutils.infrastructure.get_security_from_metadata()
synutils.infrastructure.get_metastore_from_metadata()
synutils.infrastructure.get_network_from_metadata()
synutils.infrastructure.get_global_init_script_from_metadata()
# --- Library Management ---
synutils.lib.installPyPI("pandas numpy")
synutils.lib.installCondaPackage("scipy")
# --- Kernel ---
synutils.kernel.restart_kernel()
synutils.kernel.restart_kernel(execute_cell_below=True)
# --- Spark ---
synutils.start_spark_initialization()
synutils.cancel_spark_initialization()
# --- Other ---
synutils.run_notebook("path/to/notebook.ipynb")
synutils.geo_lib.register(spark)
synutils.ray.create_cluster("template")
synutils.ray.setup_spark()

Scala

// --- Credential Store ---
synutils.credentials.create("name", Map("key" -> "val"))
synutils.credentials.read("name")                     // → SecretMap
synutils.credentials.read("name", "key")              // → SecretString
synutils.credentials.update("name", key = "k", value = "v")
synutils.credentials.update("name", updatedData = Map("k" -> "v"))
synutils.credentials.delete("name")
synutils.credentials.list()
synutils.credentials.share("user", List("read"), "name")
synutils.credentials.revoke("user", List("read"), "name")

// --- File System ---
synutils.fs.upload("/local/file", "remote/file")
synutils.fs.download("remote/file", "/local/file")
synutils.fs.uploadFolder("/local/dir/", "remote/dir/")
synutils.fs.downloadFolder("remote/dir/", "/local/dir/")
synutils.fs.list("remote/path/")
synutils.fs.isExists("remote/file")
synutils.fs.delete("remote/file")
synutils.fs.read("remote/file")                    // → Array[Byte]

// --- Infrastructure ---
infrastructure.getProviderType()
infrastructure.getConfig()
infrastructure.getStorage()
infrastructure.getSecurity()
infrastructure.getMetastore()
infrastructure.getNetwork()

// --- Spark ---
SparkInitializer("name", "KUBERNETES",
    driverMemory = Some("4g"),
    executorMemory = Some("8g"))
// `spark` is then available globally

Workflow Example

# 1. Install a package you need
synutils.lib.installPyPI("snowflake-connector-python")
# 2. Get credentials for Snowflake
sf_creds = synutils.credentials.read("snowflake_prod")
raw = sf_creds.unseal()
# 3. Connect using credentials
import snowflake.connector
conn = snowflake.connector.connect(
    user=raw["user"],
    password=raw["password"],
    account=raw["account"],
    warehouse=raw["warehouse"],
    database=raw["database"]
)
# 4. Query and load into Spark
df = spark.read.format("snowflake") \
    .options(**raw) \
    .option("query", "SELECT * FROM events") \
    .load()
# 5. Save results and upload
result = df.toPandas()
result.to_csv("/tmp/results.csv", index=False)
synutils.fs.upload("/tmp/results.csv", "my-bucket/output/results.csv")

Available Modules​

Details​

Credential Store​

create Both​

read Both​

Using Credentials with SDKs​

update Both​

delete Both​

list Both​

share Both​

revoke Both​

File System​

upload Both​

upload_folder / uploadFolder Both​

download Both​

download_folder / downloadFolder Both​

list Both​

list_file_paths Python​

is_exists / isExists Both​

delete Both​

move Python​

stream Python​

upload_stream Python​

read Scala​

Scala-Only Properties​

Infrastructure​

Library Management Python​

installPyPI​

installCondaPackage​

Kernel Utilities Python​

restart_kernel​

Spark Initialization​

Python — Manual Control​

Python — Advanced Configuration​

Scala — SparkInitializer​

SparkInitializer Parameters (Scala)​

Ray Cluster Management Python​

create_cluster​

ray_init​

setup_spark​

delete​

GeoMesa (Geospatial) Both​

Run Notebook Python​

Authentication Both​

Configuration Python​

Quick Reference​

Workflow Example​

Available Modules

Details

Credential Store

create Both

read Both

Using Credentials with SDKs

update Both

delete Both

list Both

share Both

revoke Both

File System

upload Both

upload_folder / uploadFolder Both

download Both

download_folder / downloadFolder Both

list Both

list_file_paths Python

is_exists / isExists Both

delete Both

move Python

stream Python

upload_stream Python

read Scala

Scala-Only Properties

Infrastructure

Library Management Python

installPyPI

installCondaPackage

Kernel Utilities Python

restart_kernel

Spark Initialization

Python — Manual Control

Python — Advanced Configuration

Scala — SparkInitializer

SparkInitializer Parameters (Scala)

Ray Cluster Management Python

create_cluster

ray_init

setup_spark

delete

GeoMesa (Geospatial) Both

Run Notebook Python

Authentication Both

Configuration Python

Quick Reference

Workflow Example