Lib Utilities

lib — install packages and JARs

Python — installPyPI / installCondaPackage

Method	Purpose
installPyPI(packages, acrossAllNodes=True)	pip install. Auto-downloads s3:///gs:///https:// URLs, plus .zip / .whl / .tar.gz filenames from the cluster's <bucket>/<config-folder>/deps/python/ folder. Multiple packages space-delimited.
installCondaPackage(packages, acrossAllNodes=True)	conda install (channel conda-forge). Multiple packages space-delimited.

Set acrossAllNodes=False to install on the kernel only (skip distribution to Spark worker nodes).

# 1. Regular PyPI package synutils.lib.installPyPI("requests") 
# 2. Multiple packages synutils.lib.installPyPI("requests pandas>=2.0 numpy") 
# 3. Local zip from the cluster's deps/python folder 
# (resolves to <bucket>/<config-folder>/deps/python/simple_module.zip) 
synutils.lib.installPyPI("simple_module.zip") 
# 4. Full cloud path (s3://, gs://, or https://) 
synutils.lib.installPyPI("s3://syntasa-k3s-dev/pradeepm/python_modules/simple_module.zip")
# 5. Mix everything in one call 
synutils.lib.installPyPI("requests simple_module.zip s3://syntasa-k3s-dev/pradeepm/python_modules/other.whl") 
# 6. Kernel-only install (don't distribute to Spark workers) 
synutils.lib.installPyPI("requests", acrossAllNodes=False) 
# 7. conda-forge package 
synutils.lib.installCondaPackage("scipy")

Scala — installJars

Install JARs into the running Scala kernel.

Accepts three source styles, mixed freely in a single space-delimited call:

Source	Example
Maven coordinates	"org.joda:joda-money:1.0.4"
Filename in the cluster's deps folder (<bucket>/<config-folder>/deps/jars/)	"greeterWithDollar.jar"
Full cloud-storage path	"gs://syn-400-development-kub/pradeepm/greeter.jar"

// Single source synutils.lib.installJars("org.joda:joda-money:1.0.4")
 synutils.lib.installJars("greeterWithDollar.jar") 
 synutils.lib.installJars("gs://syn-400-development-kub/pradeepm/greeter.jar") 
 
 // Multiple, space-delimited 
 synutils.lib.installJars("org.joda:joda-money:1.0.4 greeterWithDollar.jar gs://syn-400-development-kub/pradeepm/greeter.jar")