File objects Utilities

files — platform file registry

A file object in the platform pairs a base cloud-storage path with a list of files (the object's parameters). synutils.files resolves these to full cloud paths and — for DATA_FILE objects in supported formats — reads them directly into a Spark DataFrame.

Methods

Method	Purpose
get(name)	Full file object dict / Map from the API (cached per name)
getPath(name)	Full cloud paths for all files in this object — returns List[str] / List[String]
getMetadata(name)	Curated subset of metadata with renamed keys
createDataFrame(name, fileName, sep=None, header=True, inferSchema=True)	Read a registered file into a Spark DataFrame. DATA_FILE objects only; supported fileFormat: DELIMITED, JSON, PARQUET, ORC, AVRO
clearCache()	Drop cached responses (force a re-fetch on next call)

createDataFrame parameters

Parameter	Type	Default	Purpose
name	str / String	—	File object name registered in the platform
fileName	str / String	—	Specific file within the object — must match one of the entries in the object's parameters[].name list
sep	str / String	None (Py) / null (Scala) — falls back to the object's API-configured delimiter, then ","	Column separator. Used only for DELIMITED.
header	bool / Boolean	True	First row is a header. Used only for DELIMITED.
inferSchema	bool / Boolean	True	Infer column types from data. Used only for DELIMITED.

Raises:

• RuntimeError (Py) / SynUtilsException (Scala) if no SparkSession was supplied to init().
• ValueError (Py) / IllegalArgumentException (Scala) if the object is not a DATA_FILE or its fileFormat is unsupported.

Examples

Python

# 1. Inspect a file object 
info = synutils.files.get("daily_report") 
print(info["objectTypeKey"], info["fileFormat"]) 
# 2. Get full cloud paths for every file in the object 
paths = synutils.files.getPath("daily_report") 
# ['gs://my-bucket/reports/sales.csv', 'gs://my-bucket/reports/orders.csv'] 
# 3. Curated metadata subset 
meta = synutils.files.getMetadata("daily_report") 
# 4. Read one file as a DataFrame (uses the object's configured delimiter) 
df = synutils.files.createDataFrame("daily_report", "sales.csv") 
df.show(5) 
# 5. Override CSV options for this read only 
 df = synutils.files.createDataFrame( 
 "daily_report", 
 "sales.tsv", 
 sep="\t", 
 header=False, 
 inferSchema=False, ) 
# 6. JSON / PARQUET / ORC / AVRO — sep / header / inferSchema are ignored 
 events = synutils.files.createDataFrame("event_dump", "events.json") 
 sales = synutils.files.createDataFrame("sales_dump", "2024-01.parquet") 
 orders = synutils.files.createDataFrame("orders_dump", "orders.orc") 
 records = synutils.files.createDataFrame("user_records", "users.avro") 
# Note: AVRO requires the matching spark-avro JAR loaded into the runtime. 
# 7. Drop cached responses (force a re-fetch on next call) 
 synutils.files.clearCache()

Scala

// 1. Inspect a file object val info = synutils.files.get("daily_report") 
println(s"${info("objectTypeKey")} ${info("fileFormat")}") 

// 2. Full cloud paths for every file in the object 
val paths: List[String] = synutils.files.getPath("daily_report") 

// 3. Curated metadata subset 
val meta = synutils.files.getMetadata("daily_report") 
println(meta("type"), meta("fileFormat")) 

// 4. Read one file as a DataFrame (uses the object's configured delimiter) 
val df = synutils.files.createDataFrame("daily_report", "sales.csv") 
df.show(5) 

// 5. Override CSV options for this read only 
val tsv = synutils.files.createDataFrame( 
"daily_report",
 "sales.tsv", 
 sep = "\t", 
 header = false, 
 inferSchema = false ) 

// 6. JSON / PARQUET / ORC / AVRO — sep / header / inferSchema are ignored 
val events = synutils.files.createDataFrame("event_dump", "events.json") 
val sales = synutils.files.createDataFrame("sales_dump", "2024-01.parquet") 
val orders = synutils.files.createDataFrame("orders_dump", "orders.orc") 
val records = synutils.files.createDataFrame("user_records", "users.avro") 

// 7. Drop cached responses 
synutils.files.clearCache()

Tip: getPath() returns paths for all files in the object — useful when you want Spark to read everything in one go via spark.read.csv(synutils.files.getPath("daily_report")). createDataFrame() reads exactly one file at a time, identified by fileName.