泽民博客 ☰ Menu

pyspark hive

Posted by 夏泽民 Aug 4, 2020

from pyspark.sql import SparkSession, HiveContext

_SPARK_HOST = “spark://spark-master:7077” _APP_NAME = “test”

spark = SparkSession.builder.master(_SPARK_HOST).appName(_APP_NAME).getOrCreate()

data = [ (1,”3”,”145”), (1,”4”,”146”), (1,”5”,”25”), (1,”6”,”26”), (2,”32”,”32”), (2,”8”,”134”), (2,”8”,”134”), (2,”9”,”137”) ] df = spark.createDataFrame(data, [‘id’, “test_id”, ‘camera_id’])

method one，default是默认数据库的名字，write_test 是要写到default中数据表的名字

df.registerTempTable(‘test_hive’) sqlContext.sql(“create table default.write_test select * from test_hive”) https://blog.csdn.net/u011412768/article/details/93426353

阅读全文

结构体转map

Posted by 夏泽民 Aug 3, 2020

在Golang中，如何将一个结构体转成map？ 本文介绍两种方法。第一种是是使用json包解析解码编码。第二种是使用反射，使用反射的效率比较高

阅读全文

Docker组件go-event 源码学习

Posted by 夏泽民 Aug 3, 2020

go-event是一个在Docker项目中使用到的一个事件分发组件，实现了常规的广播，队列等事件分发模型，代码简洁明了，也适合初学者对Go语言的入门，对channel用来同步，通信也会加深理解。

阅读全文

chromedp

Posted by 夏泽民 Aug 3, 2020

而最近广泛使用的headless browser解决方案PhantomJS已经宣布不再继续维护，转而推荐使用headless chrome.

阅读全文

升级到php8 问题

Posted by 夏泽民 Jul 31, 2020

运行composer preg_match(): Allocation of JIT memory failed, PCRE JIT will be disabled.

阅读全文

Search

Blog Categories

jekyll (35)
web (334)
technology (2)
php (201)
linux (415)
spark (51)
deep_learning (2)
storage (290)
golang (1081)
cryptology (15)
lang (74)
docker (100)
zookeeper (11)
elasticsearch (86)
java (7)
algorithm (134)
hadoop (2)
slug (1)
architect (46)
k8s (107)
node (82)
python (18)
mysql (42)
rust (4)
elasticSearch (1)
arch (1)
Javascript (1)
kafka (2)
nginx (7)

Popular posts

Anything in here will be replaced on browsers that support the canvas element

Recent posts

Tags

This blog is maintained by 夏泽民

Get in touch with me at 465474307@qq.com

Subscribe to our mailing list

* indicates required