浅谈DataFrame和SparkSql取值误区_Python

浅谈DataFrame和SparkSql取值误区

2021-03-02 00:44silentwolfyh Python

今天小编就为大家分享一篇浅谈DataFrame和SparkSql取值误区，具有很好的参考价值，希望对大家有所帮助。一起跟随小编过来看看吧

1、dataframe返回的不是对象。

2、dataframe查出来的数据返回的是一个dataframe数据集。

3、dataframe只有遇见action的算子才能执行

4、sparksql查出来的数据返回的是一个dataframe数据集。

原始数据

scala> val parquetdf = sqlcontext.read.parquet("hdfs://hadoop14:9000/yuhui/parquet/part-r-00004.gz.parquet")

df: org.apache.spark.sql.dataframe =

[timestamp: string, appkey: string, app_version: string, channel: string, lang: string, os_type: string, os_version: string, display: string, device_type: string, mac: string, network: string, nettype: string, suuid: string, register_days:

int

, country: string, area: string, province: string, city: string, event: string, use_interval_cat: string, use_duration_cat: string, use_interval: bigint, use_duration: bigint, os_upgrade_from: string, app_upgrade_from: string, page_name: string, event_name: string, error_type: string]

浅谈DataFrame和SparkSql取值误区

代码

									package dataframe

									import org.apache.spark.sql.sqlcontext

									import org.apache.spark.{sparkconf, sparkcontext}

									/**

									 * created by yuhui on 2016/6/14.

									 */

									object dataframetest {

									 def main(args: array[string]) {

									 dataframeinto()

									 }

									 def dataframeinto() {

									 val conf = new sparkconf()

									 val sc = new sparkcontext(conf)

									 val sqlcontext = new sqlcontext(sc)

									 val df = sqlcontext.read.parquet("hdfs://hadoop14:9000/yuhui/parquet")

									 //df.map(line => printinfo(line.getstring(0)))

									 //df.foreach(line => printinfo(line.getstring(0)+" , "+line.getstring(14)+" , "+line.getstring(15)))

									 //df.select("timestamp","country","area").foreach(line=>printinfo(line.tostring))

									 df.registertemptable("infotable")

									 sqlcontext.sql("select timestamp , country , area from infotable").foreach(line=>printinfo(line.tostring))

									 }

									 def printinfo(msg: string) {println("printinfo函数-->" + msg) }

									}