|
|
|
A streaming performance measurement system
|
|
|
Streamcheck operates a unique streaming measurement service utilizing a
distributed network of audience Scanners to understand the end-user
experience. The following diagram illustrates how Streamcheck works.
A) Scanners connected to the Internet mimic audience members by
connecting to streams while quality data is collected.
B) The information collected by the Scanners is transmitted in
real-time to Streamcheck's data center.
C) The data is analyzed and, if necessary, alerts are sent
by email or pager to the appropriate people. The rules for these real-time
alerts can be set by the user.
D) Streamcheck stores all of the data in a database for further,
on-demand analysis.
E) Users can access the data through a real-time web interface or...
F) Through an XML gateway, or...
G) Through regularly scheduled email reports.
|
|
|
|
A distributed network of Scanners
|
|
|
|
At the heart of Streamcheck's operation is its network of Scanners
distributed throughout Europe, Canada and the United States.
The Scanners are connected to the public Internet
and run proprietary software that mimics the behavior of a
streaming client such as RealPlayer or Windows Media Player. Streamcheck's
Scanners can play audio or video in most popular formats delivered from a
variety of streaming servers. Future plans call for a roll-out of Scanners
in Europe, Asia and South America.
|
|
|
|
The measurement process
|
|
|
|
When a stream URL is submitted for testing, Streamcheck's back-end scheduling
application intelligently builds a profile of a stream's availability across
the Internet. This is accomplished by transmitting the URL to Scanners
situated at different points along the network. Once a Scanner receives an
assignment, it "watches" the content and reports back relevant statistics
on the user experience. Customers can view the results from these tests, in
real-time, through Streamcheck's web-based interface and analyze the
information in a variety of ways. Streamcheck complements the interface with
a rules-based email alert and notification system.
|
|
|
|
The top level Streamcheck metrics
|
|
|
|
Connection Success Rate (CSR)
This percentage represents the number of times a connection was
established with the streaming server as a percentage of total
attempted checks. While StreamQ™ captures a user perspective on
checks which successfully connect to the server, the "Success Rate"
adds the element of a users’ failure to connect altogether. Failures
to connect can arise due to several factors. These are as follows:
• Connectivity Problem- There was no available path along the Internet from the Scanner to the
address specified.
• Server Problem- The streaming server
was not active or not accepting connections.
• Logic Problem- The requested file was not found, the server didn't allow access to the file, etc.
Ongoing Streamcheck tests and results indicate that the typical connect
rate for the best streams is between 95%-99%. It is not uncommon for
popular streams to have a Connection Success Rate as low as 80%. As
surprising as this may seem, this is the current reality for the
streaming media end-user.
StreamQ™
StreamQ™ is a proprietary metric invented by Streamcheck that is rapidly
becoming an industry standard for measuring the streaming media user experience.
StreamQ™ is measured in letter grades from A+ down to F that indicate
the amount of time a user had to wait or experience interruptions (aka
"Frustration Time") versus the time actually spent enjoying the content
(aka "Play Time").
For more details on how StreamQ™ is calculated see the document
About StreamQ.
Average Bitrate (ABR)
The average bitrate of data transfer achieved offers an additional
measurement to further explain possible causes for StreamQ™ measurements
both positive and negative. While both Windows and Real media players
provide built-in bitrate measurements, their methodologies for
measurement vary. As such, Streamcheck performs its own measurement of
bitrate achieved taken during Play Time, Buffer Time and
Rebuffer Time. The reported Average Bitrate measurement consists of the
average of the sub-second interval measurements during a specific check.
|
|
|
|
Additional Steamcheck metrics
|
|
|
In addition to the above high-level summary statistics, Streamcheck also
collects and makes available a full suite of detailed check-level data
for informational and diagnostic purposes. These include transfer
protocol, server version, multiple stream source details as well as full
traceroute information from the Streamcheck Scanner to the streaming
server for each check.
For more details see the document Advanced Metrics.
|
|
|
|
Accessing test results
|
|
|
As a Streamcheck user, you have several options for accessing the
data collected on your streams.
• Web interface: The real-time web interface is the most common way to access results. It
provides several views of the data that allow quick comparisons between tasks as well as
tracking metrics over time and over locations. Click here for a tour
of the interface.
• HTML reports: Our custom reporting system makes it easy to keep an eye on your streaming
performance. Receive comprehensive, highly-configurable HTML email reports as often as you
like (hourly, daily, weekly, etc). HTML reports provide all of the metrics available from the web
interface and many more.
• Email alerts: The instant alerts immediately notify you via email/pager when your streams
are unavailable or of low quality. Find out when your quality levels are unacceptable before
your customers do by setting up user-defined rules (e.g. notify me "when streams are
unreachable 3 times in a row" or "when buffering times exceed 30 seconds").
• XML feed: Via XML you can have access to the raw data collected on your streams. Ideal
for integrating a data feed of Streamcheck's comprehensive measurements into your own
real-time reporting system.
|
