Optimal size of a Cloud Storage Fetch

One of the backbones of video streaming is the sweet spot between user connection speed, cloud hosting performance, and the window size you use to buffer and request data.

And for MegaStreams (a video streaming service company), this sweet spot is critical to their operations.

So, in order to figure that out, let’s look at what the read-performance is from GCS at various window sizes.

To figure this out, I wrote a small script which:

What we can see here is that for fetches <= 1MB we can see very consistent read costs through HTTP fetches. Once we get above that, we can see that we’re quickly bottlenecked by the HTTP protocol, rather than anything GCS related.

We can switch our test just slightly to get a better view of things. Let’s take an 8MB file, and fetch it through different sized chunks (note : smaller is better in this graph):

This graph helps us see our read performance from another perspective. As the chunk size decreases, the overhead, per-transaction increases, thus making things slower.

Combined, these two graphs illustrate something important : much like write throughput, GCS is extremely strong in terms of single stream throughput. Which means that for both uploads, and downloads, GCS is at it’s best performance when you’re favoring larger requests, around 1MB in size, otherwise, the per-transaction overhead of the requests themselves will start to slow things down.

For MegaStreams, this was a massively helpful piece of data. They now had the ability to determine what level of regional redundancy they needed for serving their video content in order to optimize the 1MB+ GCS target to improve buffer speeds. Data saves the day!

There’s some great tools to help you profile your networking perf.

Don’t forget to check you’re using the right region for your GCP instance.

And make sure you’re using the right CPU counts for your networking performance.

Related posts:

Creating passive income streams can be the difference between the live you are living now and the life you want to be living. Passive income can help you build the financial independence you’ve…

During the past few weeks, the standoff between North Korea and the United States has cooled a bit. Pyongyang has not tested more ballistic missiles or nuclear devices, and U.S. President Donald…

Helping survivors of the storm and meeting the day-to-day duties of an assistant principal means long hours. Lisa does it anyway. Hurricane Harvey slammed her community in Katy, Texas. Homes were…