Encoder Comparison NVENC x264 Quicksync QSV VP9 and AV1

x264

History

For a decade and a half, x264 has been the go-to free open-source encoder for AVC H.264 video. It is the sequel to H.263, the format encoded by DivX and Xvid. Netflix and YouTube still use H.264 to a degree and x264 is the most common encoder used by streamers.

Up until recently, the best way to get a better quality video into limited bandwidth was to use x264. Improving CPU power is the best way to improve your x264 quality-per-bitrate. There are a few reasons for this:

  • Original Kepler NVENC and Sandy Bridge QuickSync were very rudimentary in terms of quality, and only excelled in speed.
  • Maxwell and Pascal NVENC had some new options for quality. However, not all of these are available for use in OBS without complicated custom options.
  • x264 had nearly a decade of development before NVENC and QuickSync came out. x264 was very well optimised by comparison.

I’ve limited x264 encode presets to those that are reasonable on current CPUs for real-time streaming. Fortunately, going slower than Medium provides barely any benefit. Even at VerySlow the H.264 parameters find quality differently to VMAF and end up wasting bits. This is a consequence of how old the format is.

Results

x264 VMAF Scores on 2560x1440 60 fps
x264 VMAF Scores on 2560×1440 60 fps

Bear in mind that NVENC for Pascal and Maxwell cards have a “slow” and “hq” version for each bitrate. OBS cannot access slow versions of NVENC by default. So comparing for streaming quality on this test video of 1440p 60 fps, x264:

  • Beats Kepler, Maxwell and Pascal NVENC on Medium preset at 4, 6 & 8 Mbps. Medium preset x264 is Twitch’s default anchor encoder when examining alternatives.
  • Beats Kepler, Maxwell and Pascal NVENC on Faster preset at 6Mbps.
  • Always beats Kepler NVENC. Always loses to Turing NVENC, H.265, VP9 and AV1.

The situation with non-live encoding is a little different. Handbrake can encode using NVENC on the “slow” setting, which give Pascal and Maxwell a boost in quality per bitrate. However, Handbrake and FFMPEG have access to a plethora of x264 parameters. Even with Handbrake’s GUI you can access slower presets and tuning options. FFMPEG offers heaps more options for x264, there’s tonnes more customisation past simple presets. Depending on your video, you can change the parameters accordingly and optimise your x264 usage.

Conclusions for x264

When streaming, if you have a good CPU your best bet is likely x264. The downside is that if your gameplay uses a lot of CPU, it could take a hit. If you have an RTX 2080TI, 2080, 2070 or 2060 then Turing is invariably the better option. Twitch still requires H.264 video input (with some exceptions for VP9) so for now this will remain commonplace. However, YouTube already allows for streaming in VP9 depending on your channel. The near future suggests Twitch will begin allowing more codecs as video resolutions get larger.

For offline encoding you need to prioritise. If you need everything done quicker, then hardware encoder options will smash x264. Should you desire better compression, the H.265 or VP9 encoders will give better results. If you’re stuck with x264 then make sure you tune it to the video type you want.

UNREAL AUSSIES ARE ALL OVER THE WEB

HELP SUPPORT
UNREAL AUSSIES

Unreal Aussies run many events over the year to help connect and build the Australian gaming community. If you are interested in helping out in any current or future planned events or wishing to offer some more ideas for us to explore - let us know!

About Us

Unreal Aussies is for passionate gamers from all walks of life. Games come and go, but the people still remain. From meetups to tournaments, hardcore teams to charity streams, Unreal Aussies core mission is to make gaming more fun as part of a community than it can ever be alone.