|
|
General questions
|
|
|
| Q: What is the Streamcheck service?
|
| Streamcheck is a system for measuring the quality of streaming media delivery.
It focuses specifically on the end user perspective and allows you to answer the
question: "How does an average Internet user perceive my stream?".
The URL you supply will be played for one minute, at a given frequency (usually 10
times per hour) from our pool of international Scanners.
|
|
| Q: How does the Streamcheck service work?
|
| When a stream URL is submitted for testing, Streamcheck's back-end scheduling
application intelligently builds a profile of a stream's availability across the
Internet. This is accomplished by transmitting the URL to Scanners situated at
different points along the Internet. Once a Scanner receives an assignment,
it "watches" the content and reports back relevant statistics on the user
experience. Customers can view the results from these tests, in real-time, through
Streamcheck's web-based interface and analyze the information in a variety of ways.
Streamcheck complements the interface with a rules-based email alert and notification
system.
|
|
| Q: How do I sign up for the service?
|
| Click on the "Trial Signup" button on the left for a free one-week trial.
The form will allow you to submit up to 3 URLs for immediate testing.
A Streamcheck representative will contact you by email to give you a username and
password.
|
|
| Q: What is a Streamcheck Scanner?
|
| Scanners are stand-alone software applications that mimic the actions of a user
watching a stream. The Scanners are proprietary technology built on top of the
RealPlayer and the Windows Media Player, with the help of RealNetworks and Microsoft.
|
|
| Q: How many cities does Streamcheck use for its measurements?
|
| There are currently Streamcheck Scanners in 12 cities throughout the US, Canada
and Europe. However, it is important to note that geographic diversity is not the
key factor in obtaining valid delivery data. Network diversity is much more
important. Streamcheck’s engineers deploy Scanners in such a way as to efficiently
profile the public Internet at large. 10 Scanners deployed correctly can give you
a better picture (and data that is more easily analyzed) than 100 Scanners
deployed incorrectly.
For a current Scanner map, click here.
|
|
| Q: What kind of streams can Streamcheck measure?
|
| Streamcheck can measure any stream that is intended for the RealPlayer or the Windows Media Player:
Live and on-demand clips; audio, video, slideshows, and hybrids; Any bitrate.
(Also refer to question below "Which URLs can I provide to the Streamcheck System?")
|
|
| Q: How are Streamcheck results provided?
|
| You can access your full Streamcheck results via our web-based interface at
www.streamcheck.com using a personal login and password. Weekly email summaries
can be delivered to your mailbox and are customizable to your needs.
Event-triggered email alerts are also available, please see below for more information.
|
|
|
|
Technical questions
|
|
|
| Q: How can Streamcheck’s Scanners really know how an end-user experiences a stream?
|
| One of the key features of the Streamcheck system is its network of "home-based"
Scanners, which are connected to the Internet via cable modem or DSL connections
just like an audience member. These Scanners combined with our Scanners in network
centers provide a complete picture of how your stream is delivered. Monitoring
systems, which take measurements only "at the backbone" leave out a crucial
component of the last mile's effect on streaming performance.
|
|
| Q: How are the checks on my stream scheduled?
|
| Streamcheck uses a dynamic scheduling engine that decides which Scanner should
check which stream on a "just-in-time" basis. So Streamcheck Scanner #27 in
Chicago will not measure your stream at the same time, every hour.
Instead, we specify the frequency of checks per hour and then draw from our
pool of Scanners to meet that requirement. The Scanners are selected based on
availability and network diversity. One of the advantages of this approach is
that stream hosting companies cannot easily distinguish a stream request by a
Streamcheck Scanner from one requested by a user. This is necessary for true
"double-blind" testing.
|
|
| Q: Can I specify the Scanners to be used or request new locations?
|
| Yes. Although, it isn't part of our standard service, you can specify a subset
of Scanners to be used if your testing needs are large enough. We can also set up
new Scanners at the location of your choice. Both of these modifications require
some lead-time and an extra fee.
|
|
| Q: Can I find out the exact locations of Streamcheck Scanners?
|
| The exact location of our Scanners is not released to our customers. We will tell
you which Scanners will be used for your tests. And, in fact, you can view trace
routes for each individual check that was performed on your stream. (Those traces
will show every network hop from the streaming server to our Scanner.)
We guard the exact locations of our Scanner network to prevent the measurements
from being tainted. In the past, services similar to ours have been overwhelmed
by parties interested in skewing the results by "tuning" a serving network to the
measurement network. Streamcheck avoids this problem by using techniques such as
dynamic IP and frequent Scanner location changes. Any one task will have a variety
of Scanners assigned to it.
|
|
| Q: Can I customize the way Streamcheck performs its tests?
|
| Yes. Our scheduling engine is very flexible and allows us to vary almost every
aspect of the testing system. However, such changes require some lead-time and
an extra fee.
|
|
| Q: Do you provide a failure notification service?
|
| Yes. You can request email alerts based on failure criteria that you specify.
Common criteria are "3 failures in a row", "failures from 5 different cities
within one hour", etc. There is an extra fee for setting up this service.
|
|
| Q: I don't have time to analyze data; can I just get a simple number for my stream quality?
|
Streaming is itself a complicated technology that is used in many different ways by
different people. It follows that streaming metrics are similarly complicated. The
Streamcheck interface simplifies the results of your tests as much as possible, but
we believe that no single number can accurately summarize quality for "streaming" in
general. Consider the following points:
• One has to consider the particular application of the stream.
A concert webcast should have high quality audio at the expense of video. A newscast
might be the opposite.
• There are many subjective judgments to be made. Is it better to have a
quicker start-up time or higher bitrate? Is it better to have a larger frame size or higher
frame rate?
• This is a practical approach from a business perspective as well.
Will your users complain because the audio is mono vs. stereo or will they complain
because it takes 30 seconds for the stream to start? Which will cause them to leave your site?
Streamcheck provides 3 high-level metrics that summarize the performance of your
stream: Connection Success Rate, Average Bit Rate and StreamQ™.
|
|
| Q: Does Streamcheck provide subjective measurements of video quality?
|
No. After careful study and analysis we feel that attempting to provide judgments about overall
quality of a video or audio presentation raises more questions than it answers. There are many
factors that affect how your video will appear on the end-user's screen.
Optimization of the encoding process is itself a deep and complicated process. We encourage people
with a real need for quality reproduction to seek the expertise of professionals with experience in
this area. You should do extensive testing of your encoded streams prior to going live with them.
This testing can be done on local machines with the appropriate configuration. The testing of encoding
quality should not be mixed with the testing of delivery quality. Once you are satisfied with the quality
of your encoded content, and you have met your bandwidth budget, it is time to consider delivery issues.
That's where Streamcheck comes in.
We've made a conscious effort to make all of the Streamcheck metrics independent of the encoding process.
Our system is designed to help you optimize the delivery of your stream.
|
|
| Q: Do you report all the numbers from the Realplayer or Windows Media "statistics" page?
|
| No. Streaming tests generate a large amount of data. We could potentially record and display dozens of
different metrics. However, we see it as part of our role to make judgments about the value of each
metric and to process the raw information into a form that allows you to answer meaningful questions
about your streaming delivery. Packet statistics, for example, can be very misleading and does not
correlate well to streaming "experience" quantities, like buffer time. Frame rate is another metric
we support. A Streamcheck report on a RealPlayer stream can be compared side-by-side to a report
on a Windows Media stream. When there is discrepancy in how a metric is
generated or if it is not supported across the board, we don't include it in our reports.
|
|
| Q: Can I measure metafiles?
|
| Yes. If you provide Streamcheck with links to files of types like ASX, WAV, WAX, SMI and RAM. Please refer
to "Which URL's can I provide to the Streamcheck system?" for more information.
|
|
| Q: Why isn't frame rate a top level metric in the Streamcheck methodology?
|
Many customers ask about measuring frame rate. That is to be expected - it is a well-known quantity that
most people understand from their experience in traditional media and broadcasting. Unfortunately, it is a
bad metric for streaming video.
The frame rate that is reported by the streaming players is based on a data analysis of encoder output.
It is not based on actual frames displayed on your screen. The "player frame rate" depends on the codec
used and on the approach used in decoding, which varies from player to player. For example, Realplayer
will in some cases interpolate frames to provide better video. While technology like this is ultimately
beneficial to the end user, it presents problems when trying to collect consistent metrics. Should the
frame rate be measured before this post-processing or after? (The frame rate reported on Realplayer's
Statistics page is measured after, so it can sometimes be higher than the encoded frame rate!).
The only true way to measure frame rate is at the level of the video card. This can be done (and we've
done it in the past) but it introduces a very strict dependence on the hardware of the measurement
computer.
In addition, frame rate borders dangerously on being a subjective measure. Which is better: A video
stream with fewer frames per second, but sharper frames; or one with a many frames per second but
blurrier frames?
Streamcheck does, however, report frame rate for certain streams. The data is available in our 'Check Details'
section. Ask your Streamcheck representative for more information.
|
|
| Q: Can I get more technical details about the testing process?
|
| Yes. Our Streamcheck service representatives will be happy to answer additional questions. Engineers are
always available to answer the most technically demanding questions as well. Please understand that some
of our testing technology is proprietary and must be kept confidential. We apologize in advance if we
can't answer all questions.
|
|
| Q: Which URL's can I provide to the Streamcheck system?
|
The URL you provide can point to the three following files or links:
• Direct streaming file (e.g. rtsp://str.foo.com/test.rm, mms://str.foo.com/test.asf);
• Metafile (e.g. http://123.456.789/test.asx, http://123.456.789/test.smi);
• CGI processed http link (e.g.http://stream.foo.com/get_stream?stream=test&speed=22)
This includes file types such as rm, ra, rp, rt, ram, smi, asf, asx, wma, wmv, wax. There are pros and cons
to each approach. By linking directly to a stream URL, your tests won't be affected by any mistakes or
problems with metafiles, scripts, etc. You may however lose the ability to measure the effect of load
balancing or distributed server architectures.
|
|
|
|
Use of data
|
|
|
| Q: Will Streamcheck use results of my tests as part of a public study?
|
| No. We don't believe that the streaming industry is well served by public comparisons at this point.
There are too many factors and too many variables to fairly compare two streaming content companies,
even if they are in the same space.
|
|
| Q: Can we use results from Streamcheck in a press release?
|
| Yes. However, we will need to approve all press communications to ensure that our data is presented
fairly and in context. We intend for the Streamcheck brand to continue to stand for technical excellence
and impartiality. In general, we require consistent testing of at least one month on a representative
sample of streams on your site before you can quote Streamcheck scores to the press.
|
|
| Q: Can we share the results from Streamcheck with our customers/partners?
|
| Yes. Streamcheck encourages you to show your results to customers or partners as required on an ad hoc
basis. For ongoing, regular data sharing Streamcheck can provide an additional login/password or additional
copies of weekly email summaries for a small additional fee. These solutions for sharing your results are
ideal to support contractual performance clauses or service level agreements.
|
|
| Q: Do you have a program for resellers/integrators?
|
| Yes. The Streamcheck Partner Program will be unveiled shortly. It offers an advanced web interface that
allows one to create their own sub-users, schedule tasks for them and allow them to see selective results.
|
|
