- 浏览: 331053 次
- 性别:
- 来自: 北京
最新评论
-
hoey168:
请问楼主,ICE 客户端连接多个服务端,tcp -h 172. ...
ZeroC ICE之旅------负载均衡及容错 -
iOracleSun:
makeC++SharedLib 增加 -G参数即可链接成功 ...
AIX apache module问题 -
fanyonglu:
不错,讲的很细,学习中
ZeroC ICE之旅------java -
click_guobin:
...
我在深圳,每月收入850元,怎么也花不完,晒一晒我是怎么开销和投资的(zz) -
hanyu332:
引用修改%apache%/conf/httpd.conf修改为 ...
awstats日志分析小结(1)
也是采用Apache,Mysql,Python,Linux,Lighttpd等开源方案
可支持1亿/日的视频访问(100 million videos per day)
创立时间 2/2005
3/2006 3000w/日
7/2006 1亿/天
如下原文:
Update: YouTube: The Platform. YouTube adds a new rich set of APIs in order to become your video platform leader--all for free. Upload, edit, watch, search, and comment on video from your own site without visiting YouTube. Compose your site internally from APIs because you'll need to expose them later anyway.
YouTube grew incredibly fast, to over 100 million video views per day, with only a handful of people responsible for scaling the site. How did they manage to deliver all that video to all those users? And how have they evolved since being acquired by Google?
Information Sources
Platform
What's Inside?
The Stats
Recipe for handling rapid growth
while (true)
{
identify_and_fix_bottlenecks();
drink();
sleep();
notice_new_bottleneck();
}
This loop runs many times a day.
Web Servers
Video Serving
- More disks serving content which means more speed.
- Headroom. If a machine goes down others can take over.
- There are online backups.
- Apache had too much overhead.
- Uses epoll to wait on multiple fds.
- Switched from single process to multiple process configuration to handle more connections.
- CDNs replicate content in multiple places. There's a better chance of content being closer to the user, with fewer hops, and content will run over a more friendly network.
- CDN machines mostly serve out of memory because the content is so popular there's little thrashing of content into and out of memory.
- There's a long tail effect. A video may have a few plays, but lots of videos are being played. Random disks blocks are being accessed.
- Caching doesn't do a lot of good in this scenario, so spending money on more cache may not make sense. This is a very interesting point. If you have a long tail product caching won't always be your performance savior.
- Tune RAID controller and pay attention to other lower level issues to help.
- Tune memory on each machine so there's not too much and not too little.
Serving Video Key Points
Serving Thumbnails
- Lots of disk seeks and problems with inode caches and page caches at OS level.
- Ran into per directory file limit. Ext3 in particular. Moved to a more hierarchical structure. Recent improvements in the 2.6 kernel may improve Ext3 large directory handling up to 100 times, yet storing lots of files in a file system is still not a good idea.
- A high number of requests/sec as web pages can display 60 thumbnails on page.
- Under such high loads Apache performed badly.
- Used squid (reverse proxy) in front of Apache. This worked for a while, but as load increased performance eventually decreased. Went from 300 requests/second to 20.
- Tried using lighttpd but with a single threaded it stalled. Run into problems with multiprocesses mode because they would each keep a separate cache.
- With so many images setting up a new machine took over 24 hours.
- Rebooting machine took 6-10 hours for cache to warm up to not go to disk.
- Avoids small file problem because it clumps files together.
- Fast, fault tolerant. Assumes its working on a unreliable network.
- Lower latency because it uses a distributed multilevel cache. This cache works across different collocation sites.
- For more information on BigTable take a look at Google Architecture, GoogleTalk Architecture, and BigTable.
Databases
- Use MySQL to store meta data like users, tags, and descriptions.
- Served data off a monolithic RAID 10 Volume with 10 disks.
- Living off credit cards so they leased hardware. When they needed more hardware to handle load it took a few days to order and get delivered.
- They went through a common evolution: single server, went to a single master with multiple read slaves, then partitioned the database, and then settled on a sharding approach.
- Suffered from replica lag. The master is multi-threaded and runs on a large machine so it can handle a lot of work. Slaves are single threaded and usually run on lesser machines and replication is asynchronous, so the slaves can lag significantly behind the master.
- Updates cause cache misses which goes to disk where slow I/O causes slow replication.
- Using a replicating architecture you need to spend a lot of money for incremental bits of write performance.
- One of their solutions was prioritize traffic by splitting the data into two clusters: a video watch pool and a general cluster. The idea is that people want to watch video so that function should get the most resources. The social networking features of YouTube are less important so they can be routed to a less capable cluster.
- Went to database partitioning.
- Split into shards with users assigned to different shards.
- Spreads writes and reads.
- Much better cache locality which means less IO.
- Resulted in a 30% hardware reduction.
- Reduced replica lag to 0.
- Can now scale database almost arbitrarily.
Data Center Strategy
looks at different metrics to know who is closest.
Lessons Learned
- Software: DB, caching
- OS: disk I/O
- Hardware: memory, RAID
发表评论
-
Redis 2.2.0 RC1 is out
2010-12-17 10:15 1189Redis 2.2.0 RC1 新特性:很多都是我所期待的; ... -
iBATIS 3 for Java Released (BETA 1)
2009-08-09 13:52 1352A month ago iBATIS turned 7 yea ... -
Memcached 1.4.0 Release
2009-07-10 17:10 1869New Features Binary Protocol ... -
nginx-0.7.60
2009-06-16 09:01 1440Changes with nginx 0.7.60 ... -
nginx-0.7.55
2009-05-06 18:47 1103Changes with nginx 0.7.55 ... -
Open Source SSL Acceleration
2009-04-17 11:15 1692SSL acceleration is a techniq ... -
March 2009 Web Server Survey
2009-04-02 12:49 988In the March 2009 survey, we re ... -
nginx 缓存功能
2009-03-26 16:02 4386随着 nginx-0.7.44的发布,nginx的c ... -
Memcached Beta 1.3.2 Released
2009-03-12 16:21 1178We've just released memcached ... -
nginx 0.7.40
2009-03-09 17:09 992Changes with nginx 0.7.40 ... -
February 2009 Web Server Survey
2009-03-02 09:19 1035In the February 2009 survey we ... -
Handle 1 Billion Events Per Day Using a Memory Gri
2009-02-17 10:41 1007Moshe Kaplan of RockeTier shows ... -
Scaling Digg and Other Web Applications
2009-02-16 11:36 1058Joe Stump, Lead Architect at D ... -
MySpace Architecture
2009-02-13 10:39 1196Information Sources Presenta ... -
Cloud Relationship Model
2009-01-20 09:53 1115Hiya All, welcome to my first g ... -
January 2009 Web Server Survey
2009-01-19 15:33 1058In the January 2009 survey we ... -
December 2008 Web Server Survey
2008-12-25 17:47 976In the December 2008 survey, ... -
Apache 2.2.11
2008-12-15 13:24 1381Changes with Apache 2.2.11 * ... -
nginx 0.7.26
2008-12-09 12:05 1034Changes with nginx 0.7.26 ... -
Python 3.0 final released
2008-12-04 10:47 1332We are pleased to announce the ...
相关推荐
YouTube Architecture.pdf
YouTube Architecture 253 Information Sources 254 Platform 254 What's Inside? 254 The Stats 254 Recipe for handling rapid growth 255 Web Servers 255 Video Serving 256 Serving Video Key ...
YouTube技术架构详解,包括被google收购前及收购后的架构。
Google SketchUp for Site Design A Guide to Modeling Site Plans, Terrain and...Included with the book are Web links to free SketchUp exercises, models, and YouTube tutorials that expand on book concepts.
APVideoPlayer Simple and clean architecture of android YouTube video player with google sign-in and full screen.
This is a sample app demonstrating Youtube UX/UI animation using ConstraintLayout.It implements the Keyframe Animation feature in ConstrainLayout.This sample app is built on Android Architecture ...
quiz-app:一个库,反映了Caio和Mike在“如何使用Swift,TDD和Clean Architecture构建iOS应用” YouTube系列上取得的进展
It goes against the flow with the microframework concept, leaving most of the architecture choices to the developer. Through its great API, extensions, and powerful patterns, Flask helps you create ...
RealtimeDatabase-使用MVVM的干净架构。 这是一个以Kotlin为基础的应用程序构建,以示例为例,展示了如何使用三种不同的方法显示数据。 第一种方法使用回调,第二种方法使用称为... 在youtube上也可以看到:即将推出。
Youtube8m, and more. You will not only learn about the different mobile and embedded platforms supported by TensorFlow but also how to set up cloud platforms for deep learning applications. Get a ...
You will learn the performance of different DNNs on some popularly used data sets such as MNIST, CIFAR-10, Youtube8m, and more. You will not only learn about the different mobile and embedded ...
神经元网络入门视频——来源于Youtube 包括: 视频1——Data and Architecture 视频2——Forward Propagation 视频3——Gradient Descent 视频4——Backpropagation 视频5——Numerical Gradient Checking ...
:eyes:我对使用Lambda Architecture设计端到端的管道和数据转换项目感兴趣,该Lambda Architecture概括了由大数据工具组成的Streaming Micro Batching和Batching系统。 :seedling:我目前正在学习 :closed_mailbox...
Promoting your business or blog with YouTube 218 Using YouTube as a hosting and delivery platform 219 Using video delivery alternatives to embedded YouTube video 221 Employing YouTube as a search ...
搜索远程作业 搜索远程作业应用程序是一个Android应用程序,您可以在其中可以在任何地方的所有字段中搜索远程作业, 您可以轻松地搜索您的字段, 您将找到所有最新的远程作业,也可以申请直接从应用程序中找到工作。...
A Mood Analysis on Youtube Comments and a Method for Improved Social Spam Detection Chapter 44. An Improved Comfort Biased Smart Home Load Manager for Grid Connected Homes Under Direct Load Control ...
这些下一个文件夹的命名跟着youtube视频进行了一些更改。 用例:创建者的CRUD API / pkg /添加 用于添加创建者和服务以调用添加到存储库的结构 / pkg / http / rest http请求的处理程序 / pkg /存储 / pkg / ...
用户可以在youtube上查看和播放预告片 显示每部电影的评论列表 用户可以通过点击心脏图标在详细信息视图中将电影标记为收藏电影 用户可以与他们的网络共享电影预告片 离线支持:应用程序利用NetworkBoundResource ,...
It is inspired by the human brain's apparent deep (layered, hierarchical) architecture. A good overview of the theory of Deep Learning theory is [Learning Deep Architectures for AI]...
A: A+ Certification 2nd Ed For Dummies.pdf A+ Certification For Dummies 3rd Ed.chm Access 2007 AIO Desk Reference For Dummies.pdf Access 2007 Vba Programming For Dummies Feb 2007.pdf ...