Cloudinary Blog

Video Optimization, Part 2: Multi-Codec Adaptive Bitrate Streaming

Multi-Codec Adaptive Bitrate Streaming for Video Optimization

Part 1 of this series discusses the optimal way of delivering progressive video streams by taking advantage of modern, efficient codecs. That approach is ideal for short-form (under 60 seconds) videos and for videos displayed at a low resolution, such as ads and previews. But what if you're delivering longer videos with a higher resolution? No matter that you could offer them as a single file through progressive streaming, your viewers might still run into issues, such as buffering, slow start of playbacks, or even playback failures.

As sharp as modern mobile-device screens have become, they can't display all the pixels in high-resolution videos (above 1,080p). Hence, delivering a video too large for the screen is not only a waste of resources and increased bandwidth costs on your part, but also a drain on your audience’s bandwidth, lengthening the processing time on devices, consuming more battery, and rendering buffering a likely occurrence.

An effective solution is Adaptive Bitrate Streaming (ABR). Read on to find out how to combine modern codecs with ABR to deliver, through automation with Cloudinary, a lightweight and optimal experience to viewers.

Adaptive Bitrate Streaming

Adaptive Bitrate Streaming

While on YouTube, Netflix, Hulu, or a similar service, if you notice the picture quality changing from good to pixelated or vice versa, that’s ABR. Behind the scenes, instead of buffering your video when the network is slow, ABR switches dynamically to a lower-bitrate (lower-quality) version of the video that can be handled by the connection speed.

ABR also takes into account your device’s resolution. So, if you're watching a video on an iPhone X, ABR does not deliver the video’s 4K version—even if your bandwidth can accommodate such a large file—because not enough pixels are available in the display for that higher-resolution version. The magic here is that all that happens automatically with Cloudinary; you just need to configure the Cloudinary Video Player with the relevant options.

For a more in-depth discussion of ABR, see the post Plug-and-Play Adaptive Bitrate Streaming With HLS and MPEG-DASH.

Definition of Bitrate

Before delving into how to deliver an optimal adaptive experience, let's talk about the main prerequisite. Bitrate refers to the number of bits it takes to encode a one-second video. Thus, a 10-second video with a 10-MB file size has an average bitrate of 8 mbps (1 megabyte per second * 8 bits per byte). To change the video bitrate, first note this formula:

Bitrate = total pixels/second _bits/pixels * or total pixels/frame x frames/second * bits/total pixels

Hence, for a lower bitrate, you can reduce the total number of pixels by doing one of the following:

  • Reduce the resolution of the video.
  • Lower the frame rate.
  • Compress the bits per pixel with a more advanced video codec—akin to what you would do for an image.

Codecs

Part 1 elaborates on modern codecs, such as HEVC (H.265) and VP9, which render sharing pixels across frames more efficient, encoding the key frames while maintaining the visual fidelity of videos. In fact, they are about 30-50 percent more efficient than the H.264 standard. Currently, however, HEVC and VP9 do not enjoy universal browser or device support.

Not to worry. As described in Part 1, you can present your video player with multiple codec options in the adaptive bitrate manifest, which would then choose the first codec it knows how to decode. In fact, about 85 percent of Internet traffic uses one of the modern codecs, meaning that your total bandwidth would be reduced by 25 to 43 percent. You can easily calculate how cost-effective this approach would be for you.

Creation of Streaming Profiles for Your Video Files

To set up codec options, first create the streaming profiles to build the manifests. Think of a streaming profile as a list of all the variations you want to create for ABR of your videos. You can specify in a profile the aspect ratio, the resolutions, the bitrates, and any additional Cloudinary transformations for your videos, such as watermarks, subtitles, other text, and splicing in other videos.

Create a maximum of three streaming profiles, one for each codec you want to support: VP9, H.265, and H.264. Before doing that, remember that your goal is to deliver the best quality with the available bitrate given the bandwidth constraint. Also note that—

  • Since H.265 and VP9 are more efficient, you can get the same picture quality for a lower bitrate. If saving bandwidth costs is your primary concern, this is probably the way to go.
  • At a given bandwidth availability, users whose devices support those modern codecs get a more superior visual experience.

Keep in mind that the content of your video could also affect your decision. For instance, for sports or action content, you might want to maintain a higher bitrate at lower resolutions.

Creating Bitrate Profiles

Before creating the profiles, see the Cloudinary API code for creating a streaming profile for H.264: Creating Custom Streaming Profiles

Copy to clipboard
cloudinary.v2.api.create_streaming_profile('16x9_h264',
  {
    display_name: "16x9 H264 Codec",
    representations: [
    {transformation:{crop: "pad", width: 1920, aspect_ratio: "16:9", bit_rate: "8500k"}},
    {transformation:{crop: "pad", width: 1280, aspect_ratio: "16:9", bit_rate: "5500k"}},
    {transformation:{crop: "pad", width: 960, aspect_ratio: "16:9", bit_rate: "3500k"}},
    {transformation:{crop: "pad", width: 640, aspect_ratio: "16:9", bit_rate: "2000k"}},
    {transformation:{crop: "pad", width: 480, aspect_ratio: "16:9", bit_rate: "800k"}}
  ] },
  function(error, result){console.log(result);});

Copy that code, paste it into another version, and make the necessary edits for H.265, as follows:

Copy to clipboard
cloudinary.v2.api.create_streaming_profile('16x9_h265',
  {
    display_name: "16x9 H265 Codec",
    representations: [
    {transformation:{crop: "pad", video_codec: "h265", width: 1920, aspect_ratio: "16:9", bit_rate: "5500k"}},
    {transformation:{crop: "pad", video_codec: "h265", width: 1280, aspect_ratio: "16:9", bit_rate: "3500k"}},
    {transformation:{crop: "pad", video_codec: "h265", width: 960, aspect_ratio: "16:9", bit_rate: "2000k"}},
    {transformation:{crop: "pad", video_codec: "h265", width: 640, aspect_ratio: "16:9", bit_rate: "800k"}},
    {transformation:{crop: "pad", video_codec: "h265", width: 480, aspect_ratio: "16:9", bit_rate: "500k"}}
  ] },
  function(error, result){console.log(result);});

Notice the lower bitrates on H.265.

Now create a profile for VP9 by using the same bitrates you did for H.265:

Copy to clipboard
  cloudinary.v2.api.create_streaming_profile('16x9_vp9',
  {
    display_name: "16x9 vp9 Codec",
    representations: [
    {transformation:{crop: "pad", video_codec: "vp9", width: 1920, aspect_ratio: "16:9", bit_rate: "5500k"}},
    {transformation:{crop: "pad", video_codec: "vp9", width: 1280, aspect_ratio: "16:9", bit_rate: "3500k"}},
    {transformation:{crop: "pad", video_codec: "vp9", width: 960, aspect_ratio: "16:9", bit_rate: "2000k"}},
    {transformation:{crop: "pad", video_codec: "vp9", width: 640, aspect_ratio: "16:9", bit_rate: "800k"}},
    {transformation:{crop: "pad", video_codec: "vp9", width: 480, aspect_ratio: "16:9", bit_rate: "500k"}}
  ] },
  function(error, result){console.log(result);});

The above bitrates are examples only. Though they should work fine in most cases, be sure to test them out and tweak the bitrates to meet your own standards. Keep in mind that a given bitrate at a constant resolution would result in a video of higher visual quality in VP9 and H.265 than in H.264.

Now create all those versions upon upload of a new video, as follows:

CAUTION: This approach results in numerous transformations—something well worth considering because video encoding is expensive. You can offset part or even all of that cost by reducing the bandwidth, depending on the ratio of bandwidth consumption to the number of videos to be encoded. All in all, it’s cost effective for situations in which the bandwidth savings are higher than the processing cost you incur. As stated earlier, the above approach yields a 25- to 43-percent reduction in total outgoing video bandwidth. With Cloudinary, 1 TB of outgoing bandwidth equals the cost of one million transformations. You can easily calculate the total cost or cost savings.

Uploading Videos

Next, upload a new video and create the various versions. The above example would read like this:

Copy to clipboard
cloudinary.v2.uploader.upload('video-file.mp4', {resource_type: 'video', 
    eager: [{format: 'm3u8', streaming_profile: '16x9_h264'},
      {format: 'm3u8', streaming_profile: '16x9_h265'},
      {format: 'mpd', streaming_profile: '16x9_vp9'}
      ],                                   
    eager_async: true,
    eager_notification_url: `https://`mysite`/notify_endpoint', public_id:`yourPublicId`}, function(result) {console.log(result); });

Replace all the variables in bold font with the actual values. When your files are ready, you’ll get an async response to the endpoint under eager_notification_url. Remember that you’re creating a lot of files here and, even though VP9 and H.265 codecs are extremely efficient consumers of bandwidth, they take three to five times as long to encode than H.264. However, with parallel processing, the operation would take the same amount of time despite a higher CPU requirement.

When the upload is complete, the following three manifests are displayed, which you’ll need for the next step:

Copy to clipboard
https://res.cloudinary.com/demo-robert/video/upload/sp_16x9_h264/yourPublicId.m3u8
https://res.cloudinary.com/demo-robert/video/upload/sp_16x9_h265/yourPublicId.m3u8
https://res.cloudinary.com/demo-robert/video/upload/sp_16x9_vp9/yourPublicId.mpd

Now decide which manifest to pick for each visitor. (As reference, see this multi-codec HLS/DASH CodePen). Check for support for H.265 and for VP9. If neither works, fall back to the H.264 manifest.

This CodePen contains a real working example.

See the Pen Cloudinary and Multi-Codec Adaptive Bitrate Streaming by Robert Moseley (@mosedart) on CodePen.

Conclusion

At Cloudinary, we’re continuously conducting research on ABR and developing enhancements to make it easier to run codecs. Try out the process for yourself and tell us about your experience in the Comments section below. Also, do check out our HTML5 Video Player.


Further Reading on Video Transformations

Recent Blog Posts

Our $2B Valuation

By
Blackstone Growth Invests in Cloudinary

When we started our journey in 2012, we were looking to improve our lives as developers by making it easier for us to handle the arduous tasks of handling images and videos in our code. That initial line of developer code has evolved into a full suite of media experience solutions driven by a mission that gradually revealed itself over the course of the past 10 years: help companies unleash the full potential of their media to create the most engaging visual experiences.

Read more
Direct-to-Consumer E-Commerce Requires Compelling Visual Experiences

When brands like you adopt a direct–to-consumer (DTC) e-commerce approach with no involvement of retailers or marketplaces, you gain direct and timely insight into evolving shopping behaviors. Accordingly, you can accommodate shoppers’ preferences by continually adjusting your product offering and interspersing the shopping journey with moments of excitement and intrigue. Opportunities abound for you to cultivate engaging customer relationships.

Read more
Automatically Translating Videos for an International Audience

No matter your business focus—public service, B2B integration, recruitment—multimedia, in particular video, is remarkably effective in communicating with the audience. Before, making video accessible to diverse viewers involved tasks galore, such as eliciting the service of production studios to manually dub, transcribe, and add subtitles. Those operations were costly and slow, especially for globally destined content.

Read more
Cloudinary Helps Minted Manage Its Image-Generation Pipeline at Scale

Shoppers return time and again to Minted’s global online community of independent artists and designers because they know they can count on unique, statement-making products of the highest quality there. Concurrently, the visual imagery on Minted.com must do justice to the designs into which the creators have poured their hearts and souls. For Minted’s VP of Engineering David Lien, “Because we are a premium brand, we need to ensure that every single one of our product images matches the selected configuration exactly. For example, if you pick an 18x24 art print on blue canvas, we will show that exact combination on the hero images in the PDF.”

Read more
Highlights on ImageCon 2021 and a Preview of ImageCon 2022

New year, same trend! Visual media will continue to play a monumental role in driving online conversions. To keep up with visual-experience trends and best practices, Cloudinary holds an annual conference called ImageCon, a one-of-a-kind event that helps attendees create the most engaging visual experiences possible.

Read more