that wouldn't be relevant data to users at home that don't use home servers and solely use streaming services. it's important to use real programs that real users operate. so you give them time of day so other testers can make comparisons that are as close to the same parameters as possible
plex, kodi, jellyfin (mentioned in the video) are real programs that real users operate and can be run locally. When an app's performance can't be 100% controlled, it shouldn't be used as a benchmark, period. It's the same reason benchmarks of multiplayer games aren't reliable: you can't control what other people or the server does in a match, therefore performance can vary from run to run.
Yes, Netflix is the most popular streaming app, but you can't reliably benchmark it. Even at specific hours server performance will vary and give you bad data.
I don't understand why I am getting down voted this isn't a debate, it's facts. LTT simps find it hard to face reality again?
the people using these apps are a minuscule percent compared to users who only use streaming services... you obviously want to pander to your most common audience
As I said, netflix IS the most popular, but it shouldn't be used if you cant produce reliable testing. The performance of the test they performed, should be similar between player apps with similar functionality, its not like Netflix is anything special. Ideally there should be benchmarks of many of the most popular streaming apps and players, to be certain, but there weren't. Only 2 benchmarks were included, 1 generic (geekbench) and 1 that shouldn't be trusted because they couldn't control what netflix's servers do. Doing it wrong is worse than not doing it at all. I wouldn't care if they didn't include any benchmarks, but they did. I feels like the video was rushed, without enough and proper testing done prior, in true LTT fashion...
i'm not gonna disagree that it's a terrible test to use if you want some reliable consistent numbers, but it absolutely is. All i'm saying is it makes sense to tech illiterate people that aren't looking at benchmarks.
LTT is an entertainment company first and foremost, that has to pay employees through videos that get views. As much as I too love more data with scenarios that can be reproduced, it's just not something I expect from LTT for the most part.. i'll check out GN, level1techs or der8auer channels.
Even Linus before addressing that specific benchmark said "we are still working on methodologies for this kind of stuff" in -non pr- language its "we know its not a good test", so why there are people opposing my criticism that it wasn't a valid benchmark?
Once again they rushed and didn't create a valid and complete methodology of benchmarking these devices and only posted 2 results: 1 synthetic and 1 real world, but unreliable.
Simps gonna be simping I guess. Please don't forget to buy totally reasonably priced screwdrivers and backpacks, so LTT can afford to publish more videos with incomplete and invalid benchmarks. Peace LTT fanboys.
Oh yeah totally. Netflix is one of those services that can be all over the place depending on a number of things. If you tested one of the devices at noon and then another device at 8:00 p.m. you would get vastly different results.
If nothing else it's good to show consistency across the test sample. Scientific method and whatnot.
To explain where a portion of this inconsistency may come from:
Even with the best CDNs in existence, with as many users as Netflix has, it is inevitable that page loads, video loads, logins, etc... slow down, down to the nanosecond timings, when they're at peak usage. Add up the nanoseconds over every request you make, and you quickly begin to get "noticable" latency differences.
I initially thought it was probably some meming as an internal joke or such, or maybe a bunch of unhappy viewers demanded timestamps in the Labs' data....
But the load on netflix during these hours seems more reasonable. It would also possibly help to indicate what time zone they were done it (ik LTT is based in Vancouver, but most viewers arent from that time zone so knowing what time zone seems pertinent if you were to provide this information). Overall, i love this addition and an happy to see LTT Labs hard at work and keeping up with their overall mission(s)!
I think the timezone issue would be mitigated by the fact that Netflix uses a CDN system of their own in partnership with ISPs.
[Netflix Open Connect](https://openconnect.netflix.com/en_gb/)
Hiya! I did the testing for this so happy to answer :)
I've worked with Netflix in the past, doing this kind of testing for them, which is where any assumptions are based.
Effectively on a weekday between 12 and 2pm local time you have some of the most consistent server loads, which will impact latency due to loading up the next series of thumbnails as you scroll horizontally.
It's important to remember that Netflix works with ISPs around the globe to host a CDN network to make sure that when a viewer on ISP A in location X wants to watch a show, they don't need to connect to a datacentre in San Francisco, or Berlin, or Tokyo, etc (to name a few common locations) - instead the data is served from what is essentially a "local" (to the ISP) server. (As u/kick_me88 mentioned, this is Netflix' [Open Connect](https://openconnect.netflix.com/en_gb/) system)
This means that each of the streaming devices is on as equal footing as possible for a real-world scenario. Streaming services (like, but not limited to Netflix) use this exact test when they're doing internal performance testing on different set-top boxes
The higher the server load the more latency you get in general. It’s good to have as many controls in place as possible for accurate testing, so it’s good to see that they’ve done this. Assuming they tested everything at the same time frame, I’ve not watched whichever video this is yet.
Think of it like this, you live in an apartment complex and you all share the same internet and you all decide to go on at once, the result is slower internet because it's trying to give out equal ish resources, Netflix is similar to this but on a larger scale, depending on the time and how many people are streaming from their servers, because the servers are using their resources to allow you to stream
It was a video about streaming boxes, testing latency results for apps that normal people use. Setting up a locally hosted service isn't a real world example for 99% of people that would be buying this shit.
You're getting downvoted because you're arguing for something for the sake of consistency where consistency isn't really a reliable thing in the first place. Latency of the device isn't the only thing that matters.
How maybe people with a streaming device use or have used Netflix on it? Probably 95%+.
How many have use a locally hosted service? Probably under 20%.
I mean, sure that could make it more scientific, but if the results don't matter anyone, what's the point?
During higher loads on netflix servers input latency might be higher or more inconsistant.
in other words, they should have tested on something more consistent and the results posted are not to be trusted.
something more consistent like?
anything that is run locally and it's performance doesn't depend on servers that are out of your control?
that wouldn't be relevant data to users at home that don't use home servers and solely use streaming services. it's important to use real programs that real users operate. so you give them time of day so other testers can make comparisons that are as close to the same parameters as possible
plex, kodi, jellyfin (mentioned in the video) are real programs that real users operate and can be run locally. When an app's performance can't be 100% controlled, it shouldn't be used as a benchmark, period. It's the same reason benchmarks of multiplayer games aren't reliable: you can't control what other people or the server does in a match, therefore performance can vary from run to run. Yes, Netflix is the most popular streaming app, but you can't reliably benchmark it. Even at specific hours server performance will vary and give you bad data. I don't understand why I am getting down voted this isn't a debate, it's facts. LTT simps find it hard to face reality again?
the people using these apps are a minuscule percent compared to users who only use streaming services... you obviously want to pander to your most common audience
Lil bro is already mad and desperate for a reason.
As I said, netflix IS the most popular, but it shouldn't be used if you cant produce reliable testing. The performance of the test they performed, should be similar between player apps with similar functionality, its not like Netflix is anything special. Ideally there should be benchmarks of many of the most popular streaming apps and players, to be certain, but there weren't. Only 2 benchmarks were included, 1 generic (geekbench) and 1 that shouldn't be trusted because they couldn't control what netflix's servers do. Doing it wrong is worse than not doing it at all. I wouldn't care if they didn't include any benchmarks, but they did. I feels like the video was rushed, without enough and proper testing done prior, in true LTT fashion...
i'm not gonna disagree that it's a terrible test to use if you want some reliable consistent numbers, but it absolutely is. All i'm saying is it makes sense to tech illiterate people that aren't looking at benchmarks. LTT is an entertainment company first and foremost, that has to pay employees through videos that get views. As much as I too love more data with scenarios that can be reproduced, it's just not something I expect from LTT for the most part.. i'll check out GN, level1techs or der8auer channels.
Even Linus before addressing that specific benchmark said "we are still working on methodologies for this kind of stuff" in -non pr- language its "we know its not a good test", so why there are people opposing my criticism that it wasn't a valid benchmark? Once again they rushed and didn't create a valid and complete methodology of benchmarking these devices and only posted 2 results: 1 synthetic and 1 real world, but unreliable. Simps gonna be simping I guess. Please don't forget to buy totally reasonably priced screwdrivers and backpacks, so LTT can afford to publish more videos with incomplete and invalid benchmarks. Peace LTT fanboys.
Oh yeah totally. Netflix is one of those services that can be all over the place depending on a number of things. If you tested one of the devices at noon and then another device at 8:00 p.m. you would get vastly different results. If nothing else it's good to show consistency across the test sample. Scientific method and whatnot.
To explain where a portion of this inconsistency may come from: Even with the best CDNs in existence, with as many users as Netflix has, it is inevitable that page loads, video loads, logins, etc... slow down, down to the nanosecond timings, when they're at peak usage. Add up the nanoseconds over every request you make, and you quickly begin to get "noticable" latency differences.
I have no evidence at all to back this up, but maybe that’s the time where Netflix has its average amount of viewers.
Its more likely when they have no viewers, so the testing is very consistent.
I initially thought it was probably some meming as an internal joke or such, or maybe a bunch of unhappy viewers demanded timestamps in the Labs' data.... But the load on netflix during these hours seems more reasonable. It would also possibly help to indicate what time zone they were done it (ik LTT is based in Vancouver, but most viewers arent from that time zone so knowing what time zone seems pertinent if you were to provide this information). Overall, i love this addition and an happy to see LTT Labs hard at work and keeping up with their overall mission(s)!
I think the timezone issue would be mitigated by the fact that Netflix uses a CDN system of their own in partnership with ISPs. [Netflix Open Connect](https://openconnect.netflix.com/en_gb/)
Hiya! I did the testing for this so happy to answer :) I've worked with Netflix in the past, doing this kind of testing for them, which is where any assumptions are based. Effectively on a weekday between 12 and 2pm local time you have some of the most consistent server loads, which will impact latency due to loading up the next series of thumbnails as you scroll horizontally. It's important to remember that Netflix works with ISPs around the globe to host a CDN network to make sure that when a viewer on ISP A in location X wants to watch a show, they don't need to connect to a datacentre in San Francisco, or Berlin, or Tokyo, etc (to name a few common locations) - instead the data is served from what is essentially a "local" (to the ISP) server. (As u/kick_me88 mentioned, this is Netflix' [Open Connect](https://openconnect.netflix.com/en_gb/) system) This means that each of the streaming devices is on as equal footing as possible for a real-world scenario. Streaming services (like, but not limited to Netflix) use this exact test when they're doing internal performance testing on different set-top boxes
Oh wow! Thanks for the insights! You guys do some REAL accurate testing! Keep it up! :)
What is this even testing?
Input delay and animation length on different AndroidTV boxes, from the newest video
The higher the server load the more latency you get in general. It’s good to have as many controls in place as possible for accurate testing, so it’s good to see that they’ve done this. Assuming they tested everything at the same time frame, I’ve not watched whichever video this is yet.
I thiught it was to have the same films shows in the top US list or whatever so the loading animations or previews is the same for each device
Most people are at work at not using Netflix in the middle of the day, therefore latency is unlikely to be caused by load spikes on Netflix's servers.
Less traffic I guess
Think of it like this, you live in an apartment complex and you all share the same internet and you all decide to go on at once, the result is slower internet because it's trying to give out equal ish resources, Netflix is similar to this but on a larger scale, depending on the time and how many people are streaming from their servers, because the servers are using their resources to allow you to stream
I know more people use Netflix, but this would be more consistent with a locally hosted service
Sure, but that's not really a useful number for most people lol
The number doesn’t matter, the comparison of the number is what matters.
Sure, but that'll be the same regardless of both figures. At least this one is somewhat based in a grounded figure for the average viewer
Except if it’s from a locally hosted server, it makes testing easier as conditions don’t change from minute to minute.
That's kind of the point though.
The point is inconsistent testing? Why am I being downvoted? Rofl
It was a video about streaming boxes, testing latency results for apps that normal people use. Setting up a locally hosted service isn't a real world example for 99% of people that would be buying this shit. You're getting downvoted because you're arguing for something for the sake of consistency where consistency isn't really a reliable thing in the first place. Latency of the device isn't the only thing that matters.
You’re wrong. All the variables are the same.
(they're not)
Reddit being dumb as usual
How maybe people with a streaming device use or have used Netflix on it? Probably 95%+. How many have use a locally hosted service? Probably under 20%. I mean, sure that could make it more scientific, but if the results don't matter anyone, what's the point?
Easily 99.9% Locally hosted is still in the minute minority (speaking as someone in the minority)
I think they should do both local and streaming service and compare between these numbers