File Service You Can Count On
ADOTAS – Serving online advertisements places unique burdens on data center storage infrastructure. Online ad networks serve billions of small files, including text, images and video clips, while simultaneously tracking every click to fine-tune ad relevancy and ensure proper billing for ads served. Ad-serving applications must be always online to operate 24 hours per day -– downtime means lost revenue because ad networks are paid by the number of ads placed.
A leading mobile advertising network recently reported over 110 billion ad impressions served in three years. Statistics like this showcase the critical nature of storage infrastructure and the need to move beyond existing systems which cannot keep pace with traffic increases. Ad serving networks can follow basic guidelines to eliminate infrastructure bottlenecks, ensure proper click-through tracking and billing, and easily handle rapid expansion.
Unique File Serving Requirements
The direct linkage between revenue, serving small files and capturing voluminous log files poses challenges and opportunities for ad networks. Companies who maximize their ad content serving capabilities and effectively tune ad targeting schemes based on usage logs stand to capture market share from competitors and increase profitability. Ad networks handle millions to billions of ad requests, most of which are relatively small files, leading to intense pressure on file serving infrastructure. Storage systems must address key requirements to keep up with the workload.
Fast response time: Ad networks must serve ads within milliseconds upon request to meet service level agreements and bill their customers. Typically, ad networks commit to a service level agreement that ads will be delivered in a set period of time. If they cannot serve an ad within that window, they lose the opportunity to generate impressions and click-throughs, reducing revenue.
Large content libraries: Ad networks deliver unique ads or combinations of ads to each user, requiring systems to support random access across a large library of ad content. Predictable response times to randomly accessed content must decrease or remain constant as the amount of ad content scales. Fast ad serving directly from disk is critical because random access patterns to a large data set are not easily cacheable.
Easily add capacity for peak demand: Ad serving systems must scale to support peak loads. Preparing for seasonal traffic should not require months of planning, but rather enable administrators to incrementally add resources that integrate seamlessly and deliver a proportional increase in overall performance.
Transaction logging: Logging workloads are very different in comparison to small file serving. Instead of quickly retrieving ad content, ad networks drive log updates into the file serving infrastructure. Large ad networks rely on logging for ad relevancy and accurate billing. Log data comes to the system from up to hundreds or thousands of Web servers simultaneously. It is critical to handle hundreds of updates per second, often as atomic appends to prevent collisions between log writers.
High availability: Ad networks must deploy highly available infrastructure for 24/7 operation in support of partner websites and to prevent lost revenue opportunities. Storage systems that support ad serving applications must be impervious to individual disk failure or node failure. File systems must maintain ad serving capabilities throughout such disruptions.
Traditional Storage Falls Behind
Traditional storage systems, including those designed just a few years ago, did not anticipate the unique requirements of Internet-scale applications and verticals such as ad serving. Legacy file systems cannot effectively or economically meet these workload demands, and fall short in several categories.
Costly, proprietary hardware: Traditional systems often rely on custom-built, expensive hardware that has a short lifespan and high ongoing maintenance costs. Upgrades are typically full-forklift leading to premature spending.
Slow throughput at scale: Storage hardware vendors typically build larger “boxes” in a scale-up approach and place too many disk drives behind a single controller, causing performance bottlenecks that cannot economically serve many simultaneous requests. Scale-out limitations often force administrators to deploy entirely new islands of storage once the capacity of existing systems is reached. This leads to excessive spending and delays in provisioning and deploying storage.
Not built for web-scale applications: Storage designers just a few years ago could not have anticipated our present era of billions of file operations in Web-scale applications. Consequently, traditional storage systems do not efficiently handle many small files requests, they do not support effortless logging, and they do not include the integrated abilities of a key-value or object store. Ad networks can take advantage of all these functions to consolidate systems, improve reliability and reduce overall costs.
Distributed Systems Meet Ad Serving Demands
Ad serving requires a new approach to storage due to its scale and workload. Distributed systems enable ad networks to build more robust and less expensive file serving infrastructure in several ways.
Inexpensive, commodity hardware: Distributed systems can employ smart software and take advantage of standard, off-the-shelf hardware to achieve the lowest overall costs. Administrators can cut capital costs and maintain lower operating costs by avoiding proprietary hardware systems.
Consistent, low-latency ad content serving: The scale out approach balances CPU power, network bandwidth, memory and storage capacity, enabling a distributed system to consistently serve ads quickly regardless of how large the system grows. Traditional systems fail to match I/O throughput in proportion to capacity growth.
Single namespace at web-scale: Distributed systems provide a single namespace at scale, dramatically simplifying system management and ensuring that application workloads can grow without requiring fork-lift upgrades of the file serving and storage infrastructure. Traditional systems reach maximum capacity relatively quickly in Web-scale applications, leading to multiple storage islands and increased management burden.
Optimize for web workloads: A storage system optimized for Web workloads drives system consolidation, simplifies management, and yields a more cost-effective approach. Ad networks employ a combination of Web workloads including small file serving, logging, and analytics. Traditional systems are not optimized for these workloads, often resulting in costly duplication of systems and processes.
Online ad networks face challenging file serving throughput and capacity requirements. Traditional storage systems have been unable to keep pace with industry growth at price points and degrees of manageability necessary to run profitable businesses. New, innovative solutions built around distributed systems are ready to step in as Web traffic continues to grow.
Reader Comments.
No comments yet
Leave a Comment
Article Sponsor
More Features
-
Loading ...
Latest News
- What’s So Creepy About Retargeting? September 3rd 2010 ADOTAS – A recent article in The New York [...] more »
- Facebook Tests New Stalking Solution September 3rd 2010 ADOTAS – Why do I have The Police’s “Every Breath [...] more »
- Marchex Mines Calls for Analytics September 3rd 2010 ADOTAS – I met Brent Turner, executive vice president of [...] more »
- LookSmart Probes Search Marketers’ Minds September 3rd 2010 ADOTAS – What goes on in the mind of the [...] more »
- Ping a Threat to Other Social Networks? September 2nd 2010 ADOTAS – The 80s had Madonna, the 90s also had [...] more »
- Twitter Swears OAuth for User Safety, Expands Link Shortener September 2nd 2010 ADOTAS – Chatting yesterday with Arnie Gullov-Singh — CEO of [...] more »
- AOL Renews Its Google Vows September 2nd 2010 ADOTAS – Bing came a-courtin’, but AOL decided it was [...] more »
Features
- Critical Websites as Public Utilities, Part 1 September 3rd 2010
- Your Ad Where? September 2nd 2010
- Mobile Is Still About Reach September 1st 2010
- Publisher Blueprints, Part 2: Brainstorming & Accelerating Failure August 31st 2010
- Publisher Blueprints, Part 1: Pre-Defining Success August 30th 2010
Spotlight
Spotlight: Vulcan Spring Testifies to GlobalSpec’s ProwessADOTAS – With potential customers stretched across a wealth of industries, Vulcan Spring sought GlobalSpec’s expertise in reaching engineers and [...] more...
Reader Favorites
Classifieds
- Digital Strategist
- Marketing Positions Available At Snackable Media!
- Marketing Positions Available At Snackable Media!
- Interactive Media - Assistant Manager
- SEM Manager
Recent Comments
- Wesley: How do you spell 'anal retentive'?
- Miggy420: Good article, thanks for the comments I peeed a little laughing.
- Mike Hunt: Gavin Dunaway Must be high while writing this. This is less than high school level
- Russell Glass: Gavin, spot on... Every day, Amazon is serving users a phenomenal user experience by
