O'Reilly、Cloudera 主办
Make Data Work
2017年7月12-13日:培训
2017年7月13-15日:会议
北京,中国

成长的烦恼--领英大数据平台500倍扩展中应对的挑战 (Growing pains: When your big data platform grows really big)

此演讲使用中文 (This will be presented in Chinese)

Zhe Zhang (领英)
09:20–09:35 Friday, 2017-07-14
地点: 紫金大厅A(Grand Hall A)
平均得分:: ****.
(4.50, 2 次得分)

领英是全球最早应用大数据技术的公司之一。早在2008年,领英就开始在一个20台节点的机群上运行Hadoop,支持大概10个Hadoop用户。在过去的9年里,领英的大数据平台扩展了将近500倍。现在领英有超过10个Hadoop机群,总共超过1万台节点,支持超过1000个工程师,数据科学家,商业分析师运行大规模数据分析程序。数据分析工具也从最开始单一的MapReduce/Pig,发展到现在的MR,Pig,Hive,Presto,Spark SQL,Spark ML,TensorFlow,Scalding,Cascading。

在这个报告中我很高兴和大家分享一下领英大数据平台团队怎样解决大规模和高速增长带来的各种挑战。这其中有基础架构系统的规模挑战,包括Hadoop的存储和调度系统的单一主机架构。还有复杂性的规模挑战:怎么样在一个统一的平台上支持大量的各种特性的应用,从毫秒级的交互式SQL查询到运行数天的深度学习模型训练。最后,还有用户体验,系统管理,和可持续性这些围绕人的规模型挑战:怎么样在平台层面把底层系统的细节屏蔽掉,为数据和服务提供者和消费者创造一个干净,简洁,可以信赖的契约和接口。


LinkedIn was one of the earliest adopters of big data technologies. In just over 10 years, the scale of its big data ecosystem has grown drastically, from a single cluster with 20 nodes supporting 10 users in 2008 to more than 10 clusters with more than 10,000 nodes supporting more than 1,000 users in 2016. The diversity of workloads has grown even faster: the company began with only MapReduce/Pig jobs but now offers an entire marketplace with MR, Pig, Hive, Presto, Spark SQL, Spark ML, TensorFlow, and so forth.

Zhe Zhang explains how LinkedIn solves various challenges around scale.

Topics include:

  • System scalability, including resource scheduling and storage
  • How to accommodate vastly different workloads, from quick interactive SQL queries to long-running deep learning jobs
  • Human scalability, including how to hide complexity from service providers and consumers and how to architect data systems to avoid duplicate and short-termed efforts
Photo of Zhe Zhang

Zhe Zhang

领英

现任领英公司研发经理,领导核心大数据团队。该团队开发和应用HDFS,YARN,Spark,TensorFlow等开源技术,为领英公司的大数据平台提供核心的存储/计算引擎。

张喆同时还是Apache Hadoop项目的管理委员会(PMC)成员。也是Hadoop3的主要功能之一,HDFS纠删码(HDFS-EC)的作者。在加入领英之前,张喆就职于Cloudera和IBM沃森研究中心。2006年至今,在国际会议和期刊上发表论文20余篇,拥有5项美国专利。在IBM期间,获杰出技术成就奖(Outstanding Technology Achievement Award)。

Zhe Zhang is an engineering manager at LinkedIn, where he leads the Core Big Data Services team, which leverages open source technologies such as Hadoop, Spark, TensorFlow, and beyond to form the storage-compute engine of LinkedIn’s big data platform. Zhe is a PMC member of Apache Hadoop and author of HDFS erasure coding, a major feature for Hadoop 3.0. Previously, Zhe worked at Cloudera and IBM’s T. J. Watson Research Center. Zhe has over 20 research publications and 5 US patents. While at IBM, he received the Research Accomplishment Award and the Outstanding Technology Achievement Award.

联系OReillyData

关注OReillyData微信号获取最新会议信息并浏览前沿数据文章。

WeChat QRcode

 

Stay Connected Image 1
Stay Connected Image 3
Stay Connected Image 2

阅读关于大数据的最新理念。

ORB Data Site