| 入口 | SparkSession.builder.appName(...).getOrCreate() |
|---|---|
| 读 | spark.read.parquet / csv / orc |
| 写 | df.write.mode(...).partitionBy('dt').parquet(path) |
按渠道汇总示例
按渠道汇总示例
from pyspark.sql import SparkSession
spark = SparkSession.builder.appName('orders_etl').getOrCreate()
df = spark.read.parquet('s3a://lake/curated/dwd/orders/dt=2026-10-11')
(df.filter('amount > 0')
.groupBy('channel')
.sum('amount')
.write.mode('overwrite')
.parquet('s3a://lake/marts/ads/orders_channel_daily'))overwrite 目标路径写成与上游共享的目录会覆盖上游数据；不确定时先写临时目录再由调度切换。