File System Utilities

fs — direct object-store filesystem (S3 / GCS / Azure / HDFS)

The synutils.fs object auto-routes by URI scheme. Schemeless paths (bucket/key/path) are also accepted — the scheme is inferred from synutils.infrastructure.fileSystemPrefix.

Listing methods — output shape at a glance

Same shape across Python and Scala:

Method	What it returns
list(path, recursive=True)	Files → basenames (last / segment); folder markers → empty string "". With recursive=False, files → basenames; subdirectories → bucket-relative prefix path (e.g. "syn-cluster-config/deps/jars/")
ls(path)	Top-level entries as full URIs — files (s3://bucket/prefix/file.jar) + subdirectory prefixes (s3://bucket/prefix/sub/)
listRecursive(path)	All files recursively as full URIs
list_file_paths(path) (alias forlistRecursive)	All files recursively as full URIs

Methods

Method	Purpose
ls(path)	Top-level entries as full URIs
listRecursive(path)	All files recursively as full URIs
list(path, recursive=True)	Basenames (see table above)
list_file_paths(path)	Alias for listRecursive (full URIs)
exists(path)	True if file or folder-prefix exists
upload(local, remote)	Upload single file
download(remote, local)	Download single file
uploadFolder(localDir, remoteDir)	Recursive upload
downloadFolder(remoteDir, localDir)	Recursive download
copy(src, dest)	Server-side copy
move(src, dest, exclude_folders=None, rewrite_subfolders=None) (Python) move(src, dest)(Scala)	Move single file (single-object mode) or copy a directory tree with filtering (tree mode — see below)
rename(old, new)	Rename in place (same bucket/container)
delete(path)	Delete file or prefix
mkdir(path)	Create directory marker
content(path)	Read text content
read(path)	Read raw bytes (Python: bytes; Scala: Array[Byte])
head(path, maxBytes=65536)	Read first N bytes as text
writeText(path, content)	Write a text file
stream(path)	Lazy read stream for large files
uploadStream(fileObj, path)	Upload from a file-like object. Returns the provider-native upload response (boto3 put_object dict, GCS Blob, Azure upload result)
PREFIX (property)	Filesystem URI prefix (s3://, gs://, abfs://) inferred from infrastructure
getBaseDir()	User's base workspace directory: /<bucket>/syn-workspace/workspaces/<workspace>
getLocalTempDir()	Local temporary directory (/tmp or platform equivalent)

move(src, dest, exclude_folders=None, rewrite_subfolders=None) — full signature

Two modes, dispatched automatically:

• Single-object move — when src does not end with / and no filter parameters are given. Performs copy + delete on the single object (true move semantic).
• Tree mode (legacy copy-with-filter) — when src ends with / or either filter parameter is supplied. Lists every object under src, applies the filters, and copies each remaining object to the corresponding path under dest. Mirrors the legacy notebook S3FileSystem.move / GCSFileSystem.move semantics.

Heads-up: Tree mode does not delete the source. The legacy implementation was effectively a filtered tree-copy despite the move name — this implementation preserves that behaviour for backward compatibility. If you need a true tree-move, follow up with delete() on the source prefix.

Parameter	Purpose
src	Source URI. Treat as a directory prefix when it ends with / or a filter parameter is supplied.
dest	Destination URI.
exclude_folders (Python Only)	Folder names whose objects should be left in place (not copied) during a tree-mode call. Each entry is normalised to "<name>/" and substring-matched against the relative path — so exclude_folders=["logs"] excludes anything whose relative path contains "logs/".
rewrite_subfolders (Python Only)	{old: new} substring replacements applied to the destination path via str.replace.

# Single-object move (true move: copy + delete) 
synutils.fs.move("s3://bucket/a/file.csv", "s3://bucket/b/file.csv") 
# Tree copy with exclusions (legacy behaviour — does NOT delete source) 
synutils.fs.move( 
  "s3://bucket/src/", "s3://bucket/dest/", 
  exclude_folders=["logs", "temp"], 
) 
# Tree copy with subfolder rename 
synutils.fs.move( 
  "s3://bucket/src/", "s3://bucket/dest/", 
  rewrite_subfolders={"old_dir": "new_dir"}, 
)

Legacy aliases (Python only)

These names still work — they delegate to their canonical counterpart. Use the canonical name in new code.

Canonical	Legacy name
upload(local, remote)	put(local, remote)
delete(path)	rm(path)
move(src, dest)	mv(src, dest)
exists(path)	exist(path)
exists(path)	is_exists(path)
mkdir(path)	create_folder(path)
uploadFolder(src, dest)	upload_folder(src, dest)
downloadFolder(src, dest)	download_folder(src, dest)
uploadStream(file, path)	upload_stream(file, path)

Examples

Python

synutils.fs.upload("local.csv", "gs://my-bucket/remote.csv")
print(synutils.fs.exists("gs://my-bucket/remote.csv"))
# Listing — three shapes
synutils.fs.list("gs://my-bucket/data/")           # basenames (legacy AWS shape)
synutils.fs.ls("gs://my-bucket/data/")             # full URIs (top level)
synutils.fs.listRecursive("gs://my-bucket/data/")  # full URIs (recursive)
# Read raw bytes
data: bytes = synutils.fs.read("gs://my-bucket/config.bin")
# Inspect environment-derived properties
print(synutils.fs.PREFIX)            # "gs://"
print(synutils.fs.getBaseDir())      # "/my-bucket/syn-workspace/workspaces/my-ws"
print(synutils.fs.getLocalTempDir()) # "/tmp"
# upload_stream — capture the response (e.g. ETag / VersionId)
with open("/tmp/data.csv", "rb") as f:
    resp = synutils.fs.upload_stream(f, "s3://my-bucket/data.csv")
print(resp["ETag"])

Scala

synutils.fs.upload("local.csv", "gs://my-bucket/remote.csv")
println(synutils.fs.exists("gs://my-bucket/remote.csv"))

// Listing — three shapes
synutils.fs.list("gs://my-bucket/data/").foreach(println)            // basenames
synutils.fs.ls("gs://my-bucket/data/").foreach(println)              // full URIs (top level)
synutils.fs.listRecursive("gs://my-bucket/data/").foreach(println)   // full URIs (recursive)

// Read raw bytes
val data: Array[Byte] = synutils.fs.read("gs://my-bucket/config.bin")

// Inspect environment-derived properties
println(synutils.fs.PREFIX)            // "gs://"
println(synutils.fs.getBaseDir())      // "/my-bucket/syn-workspace/workspaces/my-ws"
println(synutils.fs.getLocalTempDir()) // "/tmp"