x264 has been the go-to encoder in OBS for a long time. With the release of the NVIDIA RTX 20 series, this has all changed. If you have a 2080TI, 2080, 2070, 2060 or 1660 then you are in luck. A lot of hype has been surrounding the Turing NVENC encoder and the upgrade to OBS to make better use of it.

Long story short, Turing wins.

Either pick your bitrate and scan bottom-up to see the quality you can get with each encoder. Or, pick your quality and scan left-to-right to see how much more bitrate the next encoder needs. For example, the quality from Turing “MaxQ” at 3000kbps, you would get the same from x264 “Medium” at about 3400kbps.

Source & test bench

For this test, I recorded a lossless copy of some Apex Legends gameplay at 1080p60. I ran around the training ground with some guns and shot some targets. My previous test was on Overwatch to see how codecs compared in 1440p60 but most questions I get asked are about 1080p60. Apex Legends is really popular right now among streamers, therefore it seemed appropriate to give it a shot.

Turing encodes were done on an RTX 2070. QuickSync was done on QuickSync version 6 (Coffee Lake). x264 encodes were done on an i7-8086k CPU. All were done in real-time or faster, with Turing @ 3xRT, QSV @ 2xRT and x264 between 1xRT-2xRT. x264 Slow was the slowest, averaging 70fps and dipping to about 63fps for approximately 7%-10% of the frames.

VMAF

The quality on the graph is a VMAF measurement. I used VMAF model vmaf_v0.6.1.pkl in Debian FFMPEG 4.0.1-static. There’s a brief description of what VMAF is here or you can read Netflix’s own description by the head dev Zhi Li. In summary, the higher the score, the better the video looks.

In my opinion, on this test video:

• VMAF 90+ looks as good as the best looking Apex streams I could find on Twitch today.
• VMAF 85 looks approximately as good as the top-tier streamers doing 1080p60. I found some streamers streaming VBR between 7Mbps and 10Mbps who look a little better. Usually, the channels that look good at 6Mbps are about this quality.
• VMAF 80 looks like the average to poor end of the 1080p60 streamers, usually those with bandwidth of 5Mbps or under.
• VMAF 70 is definitely on the poor quality side of things. The colours are blocky and terrain loses all its fine detail. Although you can tell what’s happening and at least it’s smooth.
• VMAF 60 is where the terrain and some models tend to block and blur into each other a lot. So much so that it can be difficult to know what a player is shooting at. At this quality, it’s definitely wise to just go down to 720p instead.

VMAF examples

Turing RTX series

Setup

There are some more test results coming that show some interesting facts about Turing. For now, I’ve gone with imitating the OBS Max Quality settings. Therefore I used this command in FFMPEG:

ffmpeg -i Original.mkv -c:v h264_nvenc -preset:v slow -profile high -level 4.2 -b:v 6000k -rc-lookahead:v 32 -an -f matroska Turing_MaxQ_6000.mkv

In this thread, dodgepong states that OBS Max Quality is the same as Quality but with 2-pass via CUDA. The FFMPEG NVENC encoder preset “Slow” is the same as “High Quality” but with 2-pass. Lookahead is set to 32 because I couldn’t find what OBS uses in the Max Quality preset so I just guessed. Spoiler alert, increasing it to 64 doesn’t change the results much anyway. I can’t figure out what the psycho visual tuning is supposed to mean, so I just left it out. Therefore the real-world use might be even better than this test. Finally, -an excludes audio from the final test files.

Results

It’s obvious from the graph that Turing NVENC beats x264 and QuickSync at any realistic bitrate. It’s not complicated and it’s not confusing. You can see similar things in another recent test bench I did with more codecs.

The drawback is that “Max Quality” in OBS uses CUDA, therefore it requires some of your GPU’s main power. If your game needs more than 90% of your GPU capacity, then I don’t recommend using it. Just use the “Quality” preset instead. It’s basically just as good. If you want more tests including this aspect, please comment below. If you can cap your frame-rate in game, then do so to secure some % of your GPU for OBS to use. Apex Legends in particular makes this tough, but each game has their own fix if you look hard enough.

Previous generations of NVENC are not as good. You can see a comparison here. Turing is the first generation that decisively beats x264.

x264

x264 presets are self-explanatory. As long as you use profile=high and level=4.2 for 1080p60 then the only difference is the preset. Interestingly “Fast” performs nearly as good as “Medium” only losing at the low bitrates but breaking even for most of the graph.

Interestingly, “Slow” loses to both Medium and Fast. I suspect there’s a combination of things in play here. First, a lot of it could just be the game. In Overwatch for example, “Slow” was much closer to Medium at higher bitrates, and beat it in lower bitrates. Second, the slower presets use older methods within H.264 of finding more quality per bit. VMAF is a much newer quality measurement, and there will be disagreements between them as to what constitutes more quality. x264 “Slow” may search harder than “Medium” for quality, but end up finding things that humans don’t really notice, and then wasting bits on those ‘improvements’.

QuickSync

Didn’t see that coming

*Updated this section on 29th March 2019

Here’s something I bet you didn’t see coming….. OBS makes great use of the NVENC encoder but only limited use of QuickSync. For some reason the drop-down box only offers 3 presets. The Kaby Lake, Coffee Lake and Whiskey Lake versions of QuickSync (v6) can 7 different presets.

The rate control of LA_ICQ is excellent for local VBR recordings. However OBS only offers 3 presets for QuickSync including “Quality”, “Balanced” and “Speed”. There are 7 presets in FFMPEG, the slowest of which is called “veryslow”. I make an assumption here, that OBS “Quality” preset is the same as the one in Handbrake. QuickSync using Handbrake preset “Quality” is the lowest scoring encoder in this test, just take another look at the graph to see it can’t compete. However using the “veryslow” preset in FFMPEG yielded results very close to, but a little BETTER THAN x264 could get.

Here’s the FFMPEG command I used:

ffmpeg -i Original.mkv -c:v h264_qsv -preset:v veryslow -profile high -level 4.2 -b:v 6000k -rdo 1 -an -f matroska Coffee_VSlow_6000.mkv

Pls improve QuickSync access in OBS!

I didn’t expect that either, but it’s there. The “veryslow” preset is currently available in FFMPEG and can be accessed through OBS using SIMPLE mode in the Output settings tab. In the field for “Custom Encoder Settings” just put “preset=veryslow”. I am unable to confirm if this works correctly since my test was done with FFMPEG.

The drawback is different to that of NVENC listed above. Since you are using an entirely different memory space it is less efficient. However it’s still capable of 1080p at 130fps so it’s fine. It’s not going to occupy any CUDA power from your main graphics card so your game can have a larger share. However OBS will STILL NEED at least some % of your main GPU’s power to render/composite the scene. Don’t make the mistake of assuming OBS needs nothing, that would be silly and logically incoherent.

The Coffee Lake (QuickSync version 6) preset “veryslow” in FFMPEG did add CPU load to my system. This didn’t go higher than 30% of a stock clocked i7-8086k. This means it may not always be useful if your game needs heavy access to your CPU. However if you can stay under 75% overall, you’re totally fine. It’s still going to use less than x264, so I would argue if you have Kaby Lake or later Intel CPU, or a Turing RTX series GPU, there aren’t many reasons to use x264 any more. It would be handy if all 7 presets appeared in the OBS drop-down box.

Please comment below if you’d like me to expand on these results with some more data.

Credits:
Featured image by Ultimatrium 

Data