SlideShare a Scribd company logo
1 of 26
by the #s
witter by the
    ing a t alk entitled “T
Giv
                   @twit  terU a t Cal.
Number    s” with
Lots o f ##s!                      ne
                 Tw itter for iPho
1m inute ago via
                     on
Retweet ed by 1 pers
What’s a Tweet?

It’s a short message that's sent through




                                           140 characters
How many are there?
How many are there?   70M
                      60M




                            Today! *




                             *off   the chart
op_oh
                                 Commons from lo
                    der Creative
       Photo used un




47M served                                               70M tweets
  per day                                                  per day
70M tweets
   per day   =   800 tweets
                 per second
Twitter by the Numbers
How big are they?


1 tweet text   =    140
                    characters
               ≈    200 bytes
800 tweets
per second       ≈       160 KB/sec
                 ≈ 9 MB/min
                 ≈ 12 GB/day
      Just tweet text!
MySQL
         Can’t generate IDs fast enough
Centralized and a single point of failure




                                    snowflake
                                    Highly available and uncoordinated (10kqps)
                                    Compatible with the           ecosystem
                                    http://github.com/twitter/snowflake
ampura
                                    Commons from ch
                       der Creative
          Photo used un




1 TB generated                                               8 TB generated
    per day                                                     per day
8 TB
per day
in total
                                                              ≈   100 MB
                                                                  per sec


  Photo used u
               nder Creative C
                              ommons from
                                            Mac Users G
                                                       uide
                                                              =   80 MB
                                                                  per sec
Where do they go?
              Followed by
  Following




                            Asymmetric Digraph
Tweets multiply
1

                   Digraph                           2
            Need to represent this

                                                         4
    1   2      3     4                    3
1




                             Matrix
2




                             Naïve implementation is not scalable
3
4
150M registered users




     2006     2008      2010
Photo used under Creative Commons from jurvetson




          Distributed graph database

flockdb   High rate of CRUD operations
          Complex set arithmetic queries
          http://github.com/twitter/flockdb
@ladygaga
mother mons†er
6.1 million followers

@BarakObama
44th President of the United States
5.3 million followers

@justinbieber
Justin Bieber
5.1 million followers

@raffi
me!
4.1 thousand followers
How do they get out?


6B API calls
    per day    ≈   70,000 calls
                   per second
REST API
         XML/JSON API over HTTP
Poll-based system / pseudo real-time




                               hosebird
                               Streaming API
                               Long poll HTTP
                               Near real-time delivery of Tweets
752%
in 2008
1358%
 in 2009
Where do we want to be?

           Today - 150M people generate ~1000 TPS

Tomorrow - we want to support half the world and all its devices

                  (5B phones and 6B people)
Real challenges in front of us
                 Real time

       Indexing, search, and analytics

             Relevance systems

              Graph databases

                   Storage

          Scalability and efficiency
Questions?   Follow me at
             twitter.com/raffi

More Related Content

What's hot

Maximizing Innovation through ChatGPT Powered Patent Analysis
Maximizing Innovation through ChatGPT Powered Patent AnalysisMaximizing Innovation through ChatGPT Powered Patent Analysis
Maximizing Innovation through ChatGPT Powered Patent AnalysisAlex G. Lee, Ph.D. Esq. CLP
 
Edgeai Engr245 2021 Lessons Learned
Edgeai Engr245 2021 Lessons LearnedEdgeai Engr245 2021 Lessons Learned
Edgeai Engr245 2021 Lessons LearnedStanford University
 
Apache Kafka as Message Queue for your microservices and other occasions
Apache Kafka as Message Queue for your microservices and other occasionsApache Kafka as Message Queue for your microservices and other occasions
Apache Kafka as Message Queue for your microservices and other occasionsMichael Reinsch
 
How do you grow when you are the leader of an organization
How do you grow when you are the leader of an organizationHow do you grow when you are the leader of an organization
How do you grow when you are the leader of an organizationKevin Goldsmith
 
Recent Trends in Deep Learning
Recent Trends in Deep LearningRecent Trends in Deep Learning
Recent Trends in Deep LearningSungjoon Choi
 
Introduction to Machine learning with Python
Introduction to Machine learning with PythonIntroduction to Machine learning with Python
Introduction to Machine learning with PythonChariza Pladin
 
High performance computing for research
High performance computing for researchHigh performance computing for research
High performance computing for researchEsteban Hernandez
 
Notes from Coursera Deep Learning courses by Andrew Ng
Notes from Coursera Deep Learning courses by Andrew NgNotes from Coursera Deep Learning courses by Andrew Ng
Notes from Coursera Deep Learning courses by Andrew NgTess Ferrandez
 
From Data to Actions and Insights at Conviva with Rui Zhang and Yan Li
From Data to Actions and Insights at Conviva with Rui Zhang and Yan Li From Data to Actions and Insights at Conviva with Rui Zhang and Yan Li
From Data to Actions and Insights at Conviva with Rui Zhang and Yan Li Databricks
 
Building Real-Time BI Systems with Kafka, Spark, and Kudu: Spark Summit East ...
Building Real-Time BI Systems with Kafka, Spark, and Kudu: Spark Summit East ...Building Real-Time BI Systems with Kafka, Spark, and Kudu: Spark Summit East ...
Building Real-Time BI Systems with Kafka, Spark, and Kudu: Spark Summit East ...Spark Summit
 
Choosing Right Garbage Collector to Increase Efficiency of Java Memory Usage
Choosing Right Garbage Collector to Increase Efficiency of Java Memory UsageChoosing Right Garbage Collector to Increase Efficiency of Java Memory Usage
Choosing Right Garbage Collector to Increase Efficiency of Java Memory UsageJelastic Multi-Cloud PaaS
 
PyBay23: Understanding LangChain Agents and Tools with Twilio (or with SMS)....
PyBay23:  Understanding LangChain Agents and Tools with Twilio (or with SMS)....PyBay23:  Understanding LangChain Agents and Tools with Twilio (or with SMS)....
PyBay23: Understanding LangChain Agents and Tools with Twilio (or with SMS)....Elizabeth (Lizzie) Siegle
 
Play! Framework for JavaEE Developers
Play! Framework for JavaEE DevelopersPlay! Framework for JavaEE Developers
Play! Framework for JavaEE DevelopersTeng Shiu Huang
 
[Outdated] Secrets of Performance Tuning Java on Kubernetes
[Outdated] Secrets of Performance Tuning Java on Kubernetes[Outdated] Secrets of Performance Tuning Java on Kubernetes
[Outdated] Secrets of Performance Tuning Java on KubernetesBruno Borges
 
HBase and HDFS: Understanding FileSystem Usage in HBase
HBase and HDFS: Understanding FileSystem Usage in HBaseHBase and HDFS: Understanding FileSystem Usage in HBase
HBase and HDFS: Understanding FileSystem Usage in HBaseenissoz
 
Kaggle presentation
Kaggle presentationKaggle presentation
Kaggle presentationHJ van Veen
 

What's hot (20)

Maximizing Innovation through ChatGPT Powered Patent Analysis
Maximizing Innovation through ChatGPT Powered Patent AnalysisMaximizing Innovation through ChatGPT Powered Patent Analysis
Maximizing Innovation through ChatGPT Powered Patent Analysis
 
Edgeai Engr245 2021 Lessons Learned
Edgeai Engr245 2021 Lessons LearnedEdgeai Engr245 2021 Lessons Learned
Edgeai Engr245 2021 Lessons Learned
 
Apache Kafka as Message Queue for your microservices and other occasions
Apache Kafka as Message Queue for your microservices and other occasionsApache Kafka as Message Queue for your microservices and other occasions
Apache Kafka as Message Queue for your microservices and other occasions
 
High–Performance Computing
High–Performance ComputingHigh–Performance Computing
High–Performance Computing
 
How do you grow when you are the leader of an organization
How do you grow when you are the leader of an organizationHow do you grow when you are the leader of an organization
How do you grow when you are the leader of an organization
 
Recent Trends in Deep Learning
Recent Trends in Deep LearningRecent Trends in Deep Learning
Recent Trends in Deep Learning
 
Introduction to Machine learning with Python
Introduction to Machine learning with PythonIntroduction to Machine learning with Python
Introduction to Machine learning with Python
 
Apache KAfka
Apache KAfkaApache KAfka
Apache KAfka
 
High performance computing for research
High performance computing for researchHigh performance computing for research
High performance computing for research
 
Notes from Coursera Deep Learning courses by Andrew Ng
Notes from Coursera Deep Learning courses by Andrew NgNotes from Coursera Deep Learning courses by Andrew Ng
Notes from Coursera Deep Learning courses by Andrew Ng
 
From Data to Actions and Insights at Conviva with Rui Zhang and Yan Li
From Data to Actions and Insights at Conviva with Rui Zhang and Yan Li From Data to Actions and Insights at Conviva with Rui Zhang and Yan Li
From Data to Actions and Insights at Conviva with Rui Zhang and Yan Li
 
Building Real-Time BI Systems with Kafka, Spark, and Kudu: Spark Summit East ...
Building Real-Time BI Systems with Kafka, Spark, and Kudu: Spark Summit East ...Building Real-Time BI Systems with Kafka, Spark, and Kudu: Spark Summit East ...
Building Real-Time BI Systems with Kafka, Spark, and Kudu: Spark Summit East ...
 
Choosing Right Garbage Collector to Increase Efficiency of Java Memory Usage
Choosing Right Garbage Collector to Increase Efficiency of Java Memory UsageChoosing Right Garbage Collector to Increase Efficiency of Java Memory Usage
Choosing Right Garbage Collector to Increase Efficiency of Java Memory Usage
 
PyBay23: Understanding LangChain Agents and Tools with Twilio (or with SMS)....
PyBay23:  Understanding LangChain Agents and Tools with Twilio (or with SMS)....PyBay23:  Understanding LangChain Agents and Tools with Twilio (or with SMS)....
PyBay23: Understanding LangChain Agents and Tools with Twilio (or with SMS)....
 
Play! Framework for JavaEE Developers
Play! Framework for JavaEE DevelopersPlay! Framework for JavaEE Developers
Play! Framework for JavaEE Developers
 
[Outdated] Secrets of Performance Tuning Java on Kubernetes
[Outdated] Secrets of Performance Tuning Java on Kubernetes[Outdated] Secrets of Performance Tuning Java on Kubernetes
[Outdated] Secrets of Performance Tuning Java on Kubernetes
 
Real-Time Event Processing
Real-Time Event ProcessingReal-Time Event Processing
Real-Time Event Processing
 
Apache Kafka
Apache KafkaApache Kafka
Apache Kafka
 
HBase and HDFS: Understanding FileSystem Usage in HBase
HBase and HDFS: Understanding FileSystem Usage in HBaseHBase and HDFS: Understanding FileSystem Usage in HBase
HBase and HDFS: Understanding FileSystem Usage in HBase
 
Kaggle presentation
Kaggle presentationKaggle presentation
Kaggle presentation
 

Viewers also liked

What we're learning about burnout and how DevOps can help
What we're learning about burnout and how DevOps can helpWhat we're learning about burnout and how DevOps can help
What we're learning about burnout and how DevOps can helpKen Mugrage
 
Big Social Data: The Spatial Turn in Big Data (Video available soon on YouTube)
Big Social Data: The Spatial Turn in Big Data (Video available soon on YouTube)Big Social Data: The Spatial Turn in Big Data (Video available soon on YouTube)
Big Social Data: The Spatial Turn in Big Data (Video available soon on YouTube)Rich Heimann
 
Re-architecting on the Fly #OReillySACon
Re-architecting on the Fly #OReillySACon Re-architecting on the Fly #OReillySACon
Re-architecting on the Fly #OReillySACon Raffi Krikorian
 
Real-time systems at Twitter (Velocity 2012)
Real-time systems at Twitter (Velocity 2012)Real-time systems at Twitter (Velocity 2012)
Real-time systems at Twitter (Velocity 2012)Raffi Krikorian
 
Aula 5 - Redes Sociais
Aula 5 - Redes SociaisAula 5 - Redes Sociais
Aula 5 - Redes SociaisRebeca Ferrari
 
Sistemas NoSQL, surgimento, características e exemplos
Sistemas NoSQL, surgimento, características e exemplosSistemas NoSQL, surgimento, características e exemplos
Sistemas NoSQL, surgimento, características e exemplosAricelio Souza
 
Scaling Twitter
Scaling TwitterScaling Twitter
Scaling TwitterBlaine
 
Deep Dive with Spark Streaming - Tathagata Das - Spark Meetup 2013-06-17
Deep Dive with Spark Streaming - Tathagata  Das - Spark Meetup 2013-06-17Deep Dive with Spark Streaming - Tathagata  Das - Spark Meetup 2013-06-17
Deep Dive with Spark Streaming - Tathagata Das - Spark Meetup 2013-06-17spark-project
 
Fitbit presentation
Fitbit presentationFitbit presentation
Fitbit presentationjryan39
 
We Are Social's Guide to Social, Digital and Mobile Around the World (Feb 2013)
We Are Social's Guide to Social, Digital and Mobile Around the World (Feb 2013)We Are Social's Guide to Social, Digital and Mobile Around the World (Feb 2013)
We Are Social's Guide to Social, Digital and Mobile Around the World (Feb 2013)We Are Social Singapore
 
Hadoop Summit Europe 2014: Apache Storm Architecture
Hadoop Summit Europe 2014: Apache Storm ArchitectureHadoop Summit Europe 2014: Apache Storm Architecture
Hadoop Summit Europe 2014: Apache Storm ArchitectureP. Taylor Goetz
 
BlueStore: a new, faster storage backend for Ceph
BlueStore: a new, faster storage backend for CephBlueStore: a new, faster storage backend for Ceph
BlueStore: a new, faster storage backend for CephSage Weil
 
Social, Digital & Mobile Around The World (January 2014)
Social, Digital & Mobile Around The World (January 2014)Social, Digital & Mobile Around The World (January 2014)
Social, Digital & Mobile Around The World (January 2014)We Are Social Singapore
 

Viewers also liked (20)

The Future of Check ins
The Future of Check insThe Future of Check ins
The Future of Check ins
 
Hacking Conway's Law
Hacking Conway's LawHacking Conway's Law
Hacking Conway's Law
 
Digital in 2017 Global Overview
Digital in 2017 Global OverviewDigital in 2017 Global Overview
Digital in 2017 Global Overview
 
What we're learning about burnout and how DevOps can help
What we're learning about burnout and how DevOps can helpWhat we're learning about burnout and how DevOps can help
What we're learning about burnout and how DevOps can help
 
Scalding @ Coursera
Scalding @ CourseraScalding @ Coursera
Scalding @ Coursera
 
Big Social Data: The Spatial Turn in Big Data (Video available soon on YouTube)
Big Social Data: The Spatial Turn in Big Data (Video available soon on YouTube)Big Social Data: The Spatial Turn in Big Data (Video available soon on YouTube)
Big Social Data: The Spatial Turn in Big Data (Video available soon on YouTube)
 
Re-architecting on the Fly #OReillySACon
Re-architecting on the Fly #OReillySACon Re-architecting on the Fly #OReillySACon
Re-architecting on the Fly #OReillySACon
 
Real-time systems at Twitter (Velocity 2012)
Real-time systems at Twitter (Velocity 2012)Real-time systems at Twitter (Velocity 2012)
Real-time systems at Twitter (Velocity 2012)
 
Aula 5 - Redes Sociais
Aula 5 - Redes SociaisAula 5 - Redes Sociais
Aula 5 - Redes Sociais
 
Sistemas NoSQL, surgimento, características e exemplos
Sistemas NoSQL, surgimento, características e exemplosSistemas NoSQL, surgimento, características e exemplos
Sistemas NoSQL, surgimento, características e exemplos
 
Scaling Twitter
Scaling TwitterScaling Twitter
Scaling Twitter
 
Deep Dive with Spark Streaming - Tathagata Das - Spark Meetup 2013-06-17
Deep Dive with Spark Streaming - Tathagata  Das - Spark Meetup 2013-06-17Deep Dive with Spark Streaming - Tathagata  Das - Spark Meetup 2013-06-17
Deep Dive with Spark Streaming - Tathagata Das - Spark Meetup 2013-06-17
 
Fitbit presentation
Fitbit presentationFitbit presentation
Fitbit presentation
 
We Are Social's Guide to Social, Digital and Mobile Around the World (Feb 2013)
We Are Social's Guide to Social, Digital and Mobile Around the World (Feb 2013)We Are Social's Guide to Social, Digital and Mobile Around the World (Feb 2013)
We Are Social's Guide to Social, Digital and Mobile Around the World (Feb 2013)
 
Digital in 2017: South America
Digital in 2017: South AmericaDigital in 2017: South America
Digital in 2017: South America
 
Presentation social network
Presentation social networkPresentation social network
Presentation social network
 
Hadoop Summit Europe 2014: Apache Storm Architecture
Hadoop Summit Europe 2014: Apache Storm ArchitectureHadoop Summit Europe 2014: Apache Storm Architecture
Hadoop Summit Europe 2014: Apache Storm Architecture
 
BlueStore: a new, faster storage backend for Ceph
BlueStore: a new, faster storage backend for CephBlueStore: a new, faster storage backend for Ceph
BlueStore: a new, faster storage backend for Ceph
 
Social, Digital & Mobile Around The World (January 2014)
Social, Digital & Mobile Around The World (January 2014)Social, Digital & Mobile Around The World (January 2014)
Social, Digital & Mobile Around The World (January 2014)
 
Big data and Hadoop
Big data and HadoopBig data and Hadoop
Big data and Hadoop
 

Similar to Twitter by the Numbers

Twitter by the Numbers (Columbia University)
Twitter by the Numbers (Columbia University)Twitter by the Numbers (Columbia University)
Twitter by the Numbers (Columbia University)Raffi Krikorian
 
Twitter for CS10 @ Berkeley (Spring 2011)
Twitter for CS10 @ Berkeley (Spring 2011)Twitter for CS10 @ Berkeley (Spring 2011)
Twitter for CS10 @ Berkeley (Spring 2011)Raffi Krikorian
 
Twitter - Guest Lecture UC Berkeley CS10 Fall 2010
Twitter - Guest Lecture UC Berkeley CS10 Fall 2010Twitter - Guest Lecture UC Berkeley CS10 Fall 2010
Twitter - Guest Lecture UC Berkeley CS10 Fall 2010Raffi Krikorian
 
Maintaining reliability in an unreliable world
Maintaining reliability in an unreliable worldMaintaining reliability in an unreliable world
Maintaining reliability in an unreliable worldJeremy Edberg
 
From incubator to exit: A brief history of Reddit, the first YCombinator success
From incubator to exit: A brief history of Reddit, the first YCombinator successFrom incubator to exit: A brief history of Reddit, the first YCombinator success
From incubator to exit: A brief history of Reddit, the first YCombinator successStartupfest
 
The Megasite: Infrastructure for Internet Scale
The Megasite: Infrastructure for Internet ScaleThe Megasite: Infrastructure for Internet Scale
The Megasite: Infrastructure for Internet Scalegoodfriday
 
Why all payments innovations are rubbish
Why all payments innovations are rubbishWhy all payments innovations are rubbish
Why all payments innovations are rubbishChris Skinner
 
The History and Possible Futures of the Internet
The History and Possible Futures of the InternetThe History and Possible Futures of the Internet
The History and Possible Futures of the InternetLarry Smarr
 
Big Data vs Data Warehousing
Big Data vs Data WarehousingBig Data vs Data Warehousing
Big Data vs Data WarehousingThomas Kejser
 
2012: The End of the World?
2012: The End of the World?2012: The End of the World?
2012: The End of the World?Saumil Shah
 
Orb - How Blockchain Industry is evolving now and future?
Orb - How Blockchain Industry is evolving now and future?Orb - How Blockchain Industry is evolving now and future?
Orb - How Blockchain Industry is evolving now and future?Orb, Inc.
 
Orb - How Blockchain Industry is evolving now and future?
Orb - How Blockchain Industry is evolving now and future?Orb - How Blockchain Industry is evolving now and future?
Orb - How Blockchain Industry is evolving now and future?Masa Nakatsu
 
The Creativity Machine
The Creativity MachineThe Creativity Machine
The Creativity MachineSal
 
Maq Software Live On Cutting Edge Dream Spark Yatra
Maq Software  Live On Cutting Edge  Dream Spark YatraMaq Software  Live On Cutting Edge  Dream Spark Yatra
Maq Software Live On Cutting Edge Dream Spark YatraMAQ Software
 
Riak at Posterous
Riak at PosterousRiak at Posterous
Riak at Posterouscapotej
 

Similar to Twitter by the Numbers (20)

Twitter by the Numbers (Columbia University)
Twitter by the Numbers (Columbia University)Twitter by the Numbers (Columbia University)
Twitter by the Numbers (Columbia University)
 
Twitter for CS10 @ Berkeley (Spring 2011)
Twitter for CS10 @ Berkeley (Spring 2011)Twitter for CS10 @ Berkeley (Spring 2011)
Twitter for CS10 @ Berkeley (Spring 2011)
 
Twitter - Guest Lecture UC Berkeley CS10 Fall 2010
Twitter - Guest Lecture UC Berkeley CS10 Fall 2010Twitter - Guest Lecture UC Berkeley CS10 Fall 2010
Twitter - Guest Lecture UC Berkeley CS10 Fall 2010
 
500Startups @ Twitter
500Startups @ Twitter500Startups @ Twitter
500Startups @ Twitter
 
Maintaining reliability in an unreliable world
Maintaining reliability in an unreliable worldMaintaining reliability in an unreliable world
Maintaining reliability in an unreliable world
 
From incubator to exit: A brief history of Reddit, the first YCombinator success
From incubator to exit: A brief history of Reddit, the first YCombinator successFrom incubator to exit: A brief history of Reddit, the first YCombinator success
From incubator to exit: A brief history of Reddit, the first YCombinator success
 
The Megasite: Infrastructure for Internet Scale
The Megasite: Infrastructure for Internet ScaleThe Megasite: Infrastructure for Internet Scale
The Megasite: Infrastructure for Internet Scale
 
Why all payments innovations are rubbish
Why all payments innovations are rubbishWhy all payments innovations are rubbish
Why all payments innovations are rubbish
 
Colorado leadership v4
Colorado leadership v4Colorado leadership v4
Colorado leadership v4
 
The History and Possible Futures of the Internet
The History and Possible Futures of the InternetThe History and Possible Futures of the Internet
The History and Possible Futures of the Internet
 
Big Data vs Data Warehousing
Big Data vs Data WarehousingBig Data vs Data Warehousing
Big Data vs Data Warehousing
 
Ngn abridged oct2010
Ngn abridged oct2010Ngn abridged oct2010
Ngn abridged oct2010
 
2012: The End of the World?
2012: The End of the World?2012: The End of the World?
2012: The End of the World?
 
Orb - How Blockchain Industry is evolving now and future?
Orb - How Blockchain Industry is evolving now and future?Orb - How Blockchain Industry is evolving now and future?
Orb - How Blockchain Industry is evolving now and future?
 
Orb - How Blockchain Industry is evolving now and future?
Orb - How Blockchain Industry is evolving now and future?Orb - How Blockchain Industry is evolving now and future?
Orb - How Blockchain Industry is evolving now and future?
 
The Creativity Machine
The Creativity MachineThe Creativity Machine
The Creativity Machine
 
IPv6 Matrix Project
IPv6 Matrix ProjectIPv6 Matrix Project
IPv6 Matrix Project
 
IPv6 Matrix Project - general presentation
IPv6 Matrix Project - general presentationIPv6 Matrix Project - general presentation
IPv6 Matrix Project - general presentation
 
Maq Software Live On Cutting Edge Dream Spark Yatra
Maq Software  Live On Cutting Edge  Dream Spark YatraMaq Software  Live On Cutting Edge  Dream Spark Yatra
Maq Software Live On Cutting Edge Dream Spark Yatra
 
Riak at Posterous
Riak at PosterousRiak at Posterous
Riak at Posterous
 

More from Raffi Krikorian

Twitter: Engineering for Real-Time (Stanford ACM 2011)
Twitter: Engineering for Real-Time (Stanford ACM 2011)Twitter: Engineering for Real-Time (Stanford ACM 2011)
Twitter: Engineering for Real-Time (Stanford ACM 2011)Raffi Krikorian
 
Securing Your Ecosystem (FOWA Las Vegas 2011)
Securing Your Ecosystem (FOWA Las Vegas 2011)Securing Your Ecosystem (FOWA Las Vegas 2011)
Securing Your Ecosystem (FOWA Las Vegas 2011)Raffi Krikorian
 
Developing for @twitterapi (Techcrunch Disrupt Hackathon)
Developing for @twitterapi (Techcrunch Disrupt Hackathon)Developing for @twitterapi (Techcrunch Disrupt Hackathon)
Developing for @twitterapi (Techcrunch Disrupt Hackathon)Raffi Krikorian
 
Twitter and the Real-Time Web
Twitter and the Real-Time WebTwitter and the Real-Time Web
Twitter and the Real-Time WebRaffi Krikorian
 
Developing for @twitterapi #hack4health
Developing for @twitterapi #hack4healthDeveloping for @twitterapi #hack4health
Developing for @twitterapi #hack4healthRaffi Krikorian
 
Intro to developing for @twitterapi (updated)
Intro to developing for @twitterapi (updated)Intro to developing for @twitterapi (updated)
Intro to developing for @twitterapi (updated)Raffi Krikorian
 
How to use Geolocation in your webapp @ FOWA Dublin 2010
How to use Geolocation in your webapp @ FOWA Dublin 2010How to use Geolocation in your webapp @ FOWA Dublin 2010
How to use Geolocation in your webapp @ FOWA Dublin 2010Raffi Krikorian
 
Intro to developing for @twitterapi
Intro to developing for @twitterapiIntro to developing for @twitterapi
Intro to developing for @twitterapiRaffi Krikorian
 
"What's Happening" to "What's Happening Here" @ Chirp
"What's Happening" to "What's Happening Here" @ Chirp"What's Happening" to "What's Happening Here" @ Chirp
"What's Happening" to "What's Happening Here" @ ChirpRaffi Krikorian
 
Handling Real-time Geostreams
Handling Real-time GeostreamsHandling Real-time Geostreams
Handling Real-time GeostreamsRaffi Krikorian
 
Adding the "Where" to the "When"
Adding the "Where" to the "When"Adding the "Where" to the "When"
Adding the "Where" to the "When"Raffi Krikorian
 
Scala + WattzOn, sitting in a tree....
Scala + WattzOn, sitting in a tree....Scala + WattzOn, sitting in a tree....
Scala + WattzOn, sitting in a tree....Raffi Krikorian
 
WattzOn Whole Earth Simulator
WattzOn Whole Earth SimulatorWattzOn Whole Earth Simulator
WattzOn Whole Earth SimulatorRaffi Krikorian
 
Broken Hearts: How Valentine's Day causes global warming
Broken Hearts: How Valentine's Day causes global warmingBroken Hearts: How Valentine's Day causes global warming
Broken Hearts: How Valentine's Day causes global warmingRaffi Krikorian
 

More from Raffi Krikorian (20)

Twitter: Engineering for Real-Time (Stanford ACM 2011)
Twitter: Engineering for Real-Time (Stanford ACM 2011)Twitter: Engineering for Real-Time (Stanford ACM 2011)
Twitter: Engineering for Real-Time (Stanford ACM 2011)
 
Securing Your Ecosystem (FOWA Las Vegas 2011)
Securing Your Ecosystem (FOWA Las Vegas 2011)Securing Your Ecosystem (FOWA Las Vegas 2011)
Securing Your Ecosystem (FOWA Las Vegas 2011)
 
Developing for @twitterapi (Techcrunch Disrupt Hackathon)
Developing for @twitterapi (Techcrunch Disrupt Hackathon)Developing for @twitterapi (Techcrunch Disrupt Hackathon)
Developing for @twitterapi (Techcrunch Disrupt Hackathon)
 
#rtgeo (Where 2.0 2011)
#rtgeo (Where 2.0 2011)#rtgeo (Where 2.0 2011)
#rtgeo (Where 2.0 2011)
 
Users and Geo
Users and GeoUsers and Geo
Users and Geo
 
Twitter and the Real-Time Web
Twitter and the Real-Time WebTwitter and the Real-Time Web
Twitter and the Real-Time Web
 
Developing for @twitterapi #hack4health
Developing for @twitterapi #hack4healthDeveloping for @twitterapi #hack4health
Developing for @twitterapi #hack4health
 
Intro to developing for @twitterapi (updated)
Intro to developing for @twitterapi (updated)Intro to developing for @twitterapi (updated)
Intro to developing for @twitterapi (updated)
 
How to use Geolocation in your webapp @ FOWA Dublin 2010
How to use Geolocation in your webapp @ FOWA Dublin 2010How to use Geolocation in your webapp @ FOWA Dublin 2010
How to use Geolocation in your webapp @ FOWA Dublin 2010
 
Intro to developing for @twitterapi
Intro to developing for @twitterapiIntro to developing for @twitterapi
Intro to developing for @twitterapi
 
Twitter API Annotations
Twitter API AnnotationsTwitter API Annotations
Twitter API Annotations
 
"What's Happening" to "What's Happening Here" @ Chirp
"What's Happening" to "What's Happening Here" @ Chirp"What's Happening" to "What's Happening Here" @ Chirp
"What's Happening" to "What's Happening Here" @ Chirp
 
Energy / Tweet
Energy / TweetEnergy / Tweet
Energy / Tweet
 
Handling Real-time Geostreams
Handling Real-time GeostreamsHandling Real-time Geostreams
Handling Real-time Geostreams
 
Adding the "Where" to the "When"
Adding the "Where" to the "When"Adding the "Where" to the "When"
Adding the "Where" to the "When"
 
What's happening here?
What's happening here?What's happening here?
What's happening here?
 
WattzOn @ ETech 2009
WattzOn @ ETech 2009WattzOn @ ETech 2009
WattzOn @ ETech 2009
 
Scala + WattzOn, sitting in a tree....
Scala + WattzOn, sitting in a tree....Scala + WattzOn, sitting in a tree....
Scala + WattzOn, sitting in a tree....
 
WattzOn Whole Earth Simulator
WattzOn Whole Earth SimulatorWattzOn Whole Earth Simulator
WattzOn Whole Earth Simulator
 
Broken Hearts: How Valentine's Day causes global warming
Broken Hearts: How Valentine's Day causes global warmingBroken Hearts: How Valentine's Day causes global warming
Broken Hearts: How Valentine's Day causes global warming
 

Recently uploaded

Building Your Own AI Instance (TBLC AI )
Building Your Own AI Instance (TBLC AI )Building Your Own AI Instance (TBLC AI )
Building Your Own AI Instance (TBLC AI )Brian Pichman
 
Meet the new FSP 3000 M-Flex800™
Meet the new FSP 3000 M-Flex800™Meet the new FSP 3000 M-Flex800™
Meet the new FSP 3000 M-Flex800™Adtran
 
VoIP Service and Marketing using Odoo and Asterisk PBX
VoIP Service and Marketing using Odoo and Asterisk PBXVoIP Service and Marketing using Odoo and Asterisk PBX
VoIP Service and Marketing using Odoo and Asterisk PBXTarek Kalaji
 
9 Steps For Building Winning Founding Team
9 Steps For Building Winning Founding Team9 Steps For Building Winning Founding Team
9 Steps For Building Winning Founding TeamAdam Moalla
 
UiPath Studio Web workshop series - Day 8
UiPath Studio Web workshop series - Day 8UiPath Studio Web workshop series - Day 8
UiPath Studio Web workshop series - Day 8DianaGray10
 
Empowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership BlueprintEmpowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership BlueprintMahmoud Rabie
 
Designing A Time bound resource download URL
Designing A Time bound resource download URLDesigning A Time bound resource download URL
Designing A Time bound resource download URLRuncy Oommen
 
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDE
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDEADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDE
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDELiveplex
 
COMPUTER 10: Lesson 7 - File Storage and Online Collaboration
COMPUTER 10: Lesson 7 - File Storage and Online CollaborationCOMPUTER 10: Lesson 7 - File Storage and Online Collaboration
COMPUTER 10: Lesson 7 - File Storage and Online Collaborationbruanjhuli
 
Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024D Cloud Solutions
 
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019IES VE
 
OpenShift Commons Paris - Choose Your Own Observability Adventure
OpenShift Commons Paris - Choose Your Own Observability AdventureOpenShift Commons Paris - Choose Your Own Observability Adventure
OpenShift Commons Paris - Choose Your Own Observability AdventureEric D. Schabell
 
Computer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and HazardsComputer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and HazardsSeth Reyes
 
Nanopower In Semiconductor Industry.pdf
Nanopower  In Semiconductor Industry.pdfNanopower  In Semiconductor Industry.pdf
Nanopower In Semiconductor Industry.pdfPedro Manuel
 
Machine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdfMachine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdfAijun Zhang
 
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCostKubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCostMatt Ray
 
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...Aggregage
 
Building AI-Driven Apps Using Semantic Kernel.pptx
Building AI-Driven Apps Using Semantic Kernel.pptxBuilding AI-Driven Apps Using Semantic Kernel.pptx
Building AI-Driven Apps Using Semantic Kernel.pptxUdaiappa Ramachandran
 
Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)Commit University
 
Cybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptxCybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptxGDSC PJATK
 

Recently uploaded (20)

Building Your Own AI Instance (TBLC AI )
Building Your Own AI Instance (TBLC AI )Building Your Own AI Instance (TBLC AI )
Building Your Own AI Instance (TBLC AI )
 
Meet the new FSP 3000 M-Flex800™
Meet the new FSP 3000 M-Flex800™Meet the new FSP 3000 M-Flex800™
Meet the new FSP 3000 M-Flex800™
 
VoIP Service and Marketing using Odoo and Asterisk PBX
VoIP Service and Marketing using Odoo and Asterisk PBXVoIP Service and Marketing using Odoo and Asterisk PBX
VoIP Service and Marketing using Odoo and Asterisk PBX
 
9 Steps For Building Winning Founding Team
9 Steps For Building Winning Founding Team9 Steps For Building Winning Founding Team
9 Steps For Building Winning Founding Team
 
UiPath Studio Web workshop series - Day 8
UiPath Studio Web workshop series - Day 8UiPath Studio Web workshop series - Day 8
UiPath Studio Web workshop series - Day 8
 
Empowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership BlueprintEmpowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership Blueprint
 
Designing A Time bound resource download URL
Designing A Time bound resource download URLDesigning A Time bound resource download URL
Designing A Time bound resource download URL
 
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDE
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDEADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDE
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDE
 
COMPUTER 10: Lesson 7 - File Storage and Online Collaboration
COMPUTER 10: Lesson 7 - File Storage and Online CollaborationCOMPUTER 10: Lesson 7 - File Storage and Online Collaboration
COMPUTER 10: Lesson 7 - File Storage and Online Collaboration
 
Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024
 
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
 
OpenShift Commons Paris - Choose Your Own Observability Adventure
OpenShift Commons Paris - Choose Your Own Observability AdventureOpenShift Commons Paris - Choose Your Own Observability Adventure
OpenShift Commons Paris - Choose Your Own Observability Adventure
 
Computer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and HazardsComputer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and Hazards
 
Nanopower In Semiconductor Industry.pdf
Nanopower  In Semiconductor Industry.pdfNanopower  In Semiconductor Industry.pdf
Nanopower In Semiconductor Industry.pdf
 
Machine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdfMachine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdf
 
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCostKubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
 
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
 
Building AI-Driven Apps Using Semantic Kernel.pptx
Building AI-Driven Apps Using Semantic Kernel.pptxBuilding AI-Driven Apps Using Semantic Kernel.pptx
Building AI-Driven Apps Using Semantic Kernel.pptx
 
Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)
 
Cybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptxCybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptx
 

Twitter by the Numbers

  • 2. witter by the ing a t alk entitled “T Giv @twit terU a t Cal. Number s” with Lots o f ##s! ne Tw itter for iPho 1m inute ago via on Retweet ed by 1 pers
  • 3. What’s a Tweet? It’s a short message that's sent through 140 characters
  • 4. How many are there?
  • 5. How many are there? 70M 60M Today! * *off the chart
  • 6. op_oh Commons from lo der Creative Photo used un 47M served 70M tweets per day per day
  • 7. 70M tweets per day = 800 tweets per second
  • 9. How big are they? 1 tweet text = 140 characters ≈ 200 bytes
  • 10. 800 tweets per second ≈ 160 KB/sec ≈ 9 MB/min ≈ 12 GB/day Just tweet text!
  • 11. MySQL Can’t generate IDs fast enough Centralized and a single point of failure snowflake Highly available and uncoordinated (10kqps) Compatible with the ecosystem http://github.com/twitter/snowflake
  • 12. ampura Commons from ch der Creative Photo used un 1 TB generated 8 TB generated per day per day
  • 13. 8 TB per day in total ≈ 100 MB per sec Photo used u nder Creative C ommons from Mac Users G uide = 80 MB per sec
  • 14. Where do they go? Followed by Following Asymmetric Digraph
  • 16. 1 Digraph 2 Need to represent this 4 1 2 3 4 3 1 Matrix 2 Naïve implementation is not scalable 3 4
  • 17. 150M registered users 2006 2008 2010
  • 18. Photo used under Creative Commons from jurvetson Distributed graph database flockdb High rate of CRUD operations Complex set arithmetic queries http://github.com/twitter/flockdb
  • 19. @ladygaga mother mons†er 6.1 million followers @BarakObama 44th President of the United States 5.3 million followers @justinbieber Justin Bieber 5.1 million followers @raffi me! 4.1 thousand followers
  • 20. How do they get out? 6B API calls per day ≈ 70,000 calls per second
  • 21. REST API XML/JSON API over HTTP Poll-based system / pseudo real-time hosebird Streaming API Long poll HTTP Near real-time delivery of Tweets
  • 24. Where do we want to be? Today - 150M people generate ~1000 TPS Tomorrow - we want to support half the world and all its devices (5B phones and 6B people)
  • 25. Real challenges in front of us Real time Indexing, search, and analytics Relevance systems Graph databases Storage Scalability and efficiency
  • 26. Questions? Follow me at twitter.com/raffi