Export PostgreSQL to S3 Compatible in Parquet
Fast, parallel data export with zero intermediate storage
Terminal
.\FastBCP.exe `
--connectiontype "pgsql" `
--server "host.domain | host.domain,port | host.domain,port/service" `
--database "tpch" `
--trusted `
--sourceschema "tpch10" `
--sourcetable "orders" `
--query "SELECT * FROM tpch10.orders" `
--directory "s3://rootbucket/fastbcpexport/raw/{sourcedatabase}/{sourceschema}" `
--fileoutput "{sourcetable}.parquet" `
--method "Ntile" `
--distributekeycolumn "o_orderkey" `
--paralleldegree -2 `
--merge false `
--runid "runidfromcaller"Source - PostgreSQL
PostgreSQL is a powerful and robust open-source relational database. FastBCP optimizes exports from PostgreSQL using its native connector for excellent performance.
Features:
- •Direct streaming read from database
- •Full support for PostgreSQL data types
- •Secure SSL connections
Parallel Method - Ntile
Divides data into N equal partitions based on a numeric column.
Requirement: Requires a numeric distribution column
Available parallel methods with PostgreSQL:
Output Format - Apache Parquet
Parquet is a columnar file format optimized for analytical processing. FastBCP exports to Parquet with compression and type preservation, ideal for data lake architectures.
Features:
- •Columnar storage for efficient analytics
- •Integrated compression (Snappy, Gzip)
- •Full data type preservation
- •Optimized for big data tools (Spark, Hadoop)
Destination - S3 Compatible Storage
FastBCP supports S3-compatible storage services like MinIO, Wasabi, and others with the same high-performance multipart upload capabilities.
Storage Type:
Cloud Object Storage
Features:
- •S3 API compatibility
- •Parallel multipart upload
- •Custom endpoint configuration
- •Works with MinIO, Wasabi, etc.
