VRguy podcast Episode 21: Tero Sarkkinen, CEO of Basmark

The VRguy's podcast

My guest today is Tero Sarkkinen, CEO of Basemark. This episode was recorded on Nov 22nd, 2016.

Tero and I talk about performance measurement in VR: what can be measured, how it’s measured and what future measurements are coming

http://sensics.com/wp-content/uploads/2016/11/Tero_sarkkinen_22_Nov_16.mp3

Interview transcript

Yuval Boger (VRguy): Hello, Tero and welcome to the podcast.

Tero: Hey, thank you well. I’m really glad to be here.

VRguy: So who are you and what do you do?

Tero: So my name is Tero Sarkkinen. I’m from Finland originally. I’m a serial entrepreneur and I spent my career for seventeen years in computer graphics.

VRguy: Excellent. What does your current company do?

Tero: Currently there are sort of two companies, one is RightWare which I started seven years ago, and that supplying computer graphics to digital instrument clusters in automotive world, but then more importantly over here is Basemark. That’s developing testing tools for VR and also rendering engines for mission critical VR.

VRguy: And before VR you guys were doing testing for non VR applications, right? CPUs, graphics cards ..
Tero: Correct. So basically I started my career, then it was PC graphics back in the, seventeen years ago, then over 10 years ago we moved smartphones and tablets, and they started to have graphics. For over a year we’ve developed the first VR testing and the first one will be testing the PC based VR.

VRguy: Excellent, and who are your customers? Are they the graphics vendors, the device vendors, the game developers, who are the customers?

Tero: During the development, it’s testing standard the customers are the semi-conductor guys. AMD, Nvidia, Intel, and so forth. Right now we are going towards launching the product finally, and once it’s out there, then the customer base will expand to technology, media, then PC manufacturers, HMD manufacturers, and also the business customers who are thinking “Okay, so what kind of PC will I need for my business applications and what is the optimal HMD for the PC that I have?”

VRguy: Today in VR, I think you have a version of latency tests. Could you explain what are you measuring and to the extent you’re willing to, how are you measuring it?

Tero: So we have patent-pending innovation, it’s a device we called “VR Track” and it’s basically a device that enables you to measure, very accurately, latency of the HMD. Now when we talk about latency, there’s different levels of latency. There’s the application to photon and then the next level is that motion to photo, and another level is the controller to photon. So what we got ready is the application to photon device, we’ve already started the selling of this actually and the technology that enables game developers, hardware assistant developers, to measure the latency with the application they’re developing. So the device is fairly simple, but it’s the software on the algorithm on the background tab that’s the real innovation. So, basically use that power measurement device inside of the HMD glasses, then you start on your workstation our testing software, and enable to measure the latency between when the signal was received, to when the actual frame appeared on the HMD.

VRguy: So you mentioned application to photon, and motion to photon, and I think also controller to photon. Could you explain how you define the differences between all three?

Tero: Yeah. So to start from the application to photon. So the time measured from the time when on your rendering system, so be it your PC, your laptop, or if you have an embedded HMD. Basically at the simulation, your software there is a command that says “update the world” and then start the timer from that and then measure the time lapses until that new screen is actually shown on your HMD. There are many steps: there’s the graphics driver, and then the HMD sdk that are in the way, and then also transferring the signal of the pixels through the HMD itself.

So this one pretty good measurement. So you can distinguish which rendering system is better than the other, and which HMD is better than the other for a given rendering system. Of course, we also need to be able to take into account the HMD’s sensors, the gyros, and how long of time elapses when I move my head until the screen is refreshed to the proper screen of the HMD. So that’s the next level and as I said, we are developing already … Well we got a really well working solution in our labs, and we are just reading it for commercial release later on.

VRguy: So motion to photon as you define it would be, usually greater than application to photon, right?
Tero: It gives you more information, but of course it’s really good to be able to also isolate, or get into more granular level into latency measurement, so that you know which part, or where in this chain I have the most latency, so that if you’re actually developing that system or solution you know which part to optimize.

VRguy: Now motion to photon depends on motion, so do you have a robotic arm, do you have the user moving the HMD, how do you induce motion?

Tero: We haven’t yet disclosed the method for motion to photo but it’s based on user moving the device, no need for robotic arm. We are very excited about this, because this means that we can enable pretty much anybody on the planet to do these measurements on their own workstation.

VRguy: Do you also account for the scan direction of the screens, I guess if you have a photo sensor inside the HMD then the scan direction would matter, unless the photo sensor is in the middle of the eye perhaps.

Tero: Our current solution for everybody, is exactly that, the photo sensor at the middle. So if you need more fine grained, like the scan-line then a laboratory environment is the thing for you.

VRguy: Now, there are other things that I think people want to measure in VR, for instance accuracy of the tracking solution, or even response time and accuracy of controllers. What are your plans along those lines?

Tero: The controllers are certainly an important one, and we already have some prototypes in that area, and it seems that our methodology we can apply also two sensors. More of that a bit later.

VRguy: Okay. Do you find that customers or vendors argue with your results? That they say “Oh, you didn’t measure it correctly” or “We don’t agree” or “We have measured something else”

Tero: Always. There’s questions, how is this really a valid methodology. So far we haven’t encountered a question that we could of explained. So everybody so far has accepted our methodology and said “Yes, this is true”.

VRguy: And, it is not just for VR, right? I guess you’re referring also to other graphics measurements.
Tero: Yes, this is the main work of developing a benchmark is to basically have the ability to develop a workload or a test that then says, let’s take a company like AMD or Nvidia they have a feud competition between each other and the ability for us to develop a test measurement product that the both can say “Okay, this is a fair test”. So that’s really the secret to the success in the benchmark industry, being able to do that.

VRguy: So when I think about PC Benchmarking, you could say “Oh, how fast is your processor” and “How fast is your disk and graphics card” and other components of the system. Do you also make recommendations with regards to how fast you need it to be for a particular application? If I wanted to watch 360 degree video on an HMD, maybe I don’t need the same performance as a first person shooter in gaming.

Tero: VR score, first VR testing product is the first of our benchmarks that also has this sort of readiness meter. So it can tell whether your system is good enough entry level VR, or is it not VR ready, and then it will also show you how long you are in the enabling great VR experience.

VRguy: Then there’s likely going to be some online database of scores when new products come out?
Tero: Ranking list for PCs graphics cards, HMDs, and so forth. Yep, absolutely. That’s the key to enable or to give this information to a lot of people who are interested in performance.

VRguy: How about eye tracking? Is that something that you’re thinking about? Measuring the performance of eye-trackers, seems to be a sensor that’s going to become popular.

Tero: Yes. Particularly in mobile VR solutions, this is our key technology. By the way, we are already quite far along in developing a mobile embedded VR score, and yes we are thinking and we are researching and seeing how we can take a measure basically for those techniques and technologies.

VRguy: Do you also think about optical quality. I mean when we compare HMDs or when we compare optical designs, there are the design parameters that we sometimes look at, you know, distortion or spot size or chromatic aberration, but then when people compare off the shelf products they talk about, God rays and they talk about pupil swim and other parameters. Is there any tests that you’re planning to measure field of view, to measure distortion, to give us a user extensive how good the visual quality will be?
Tero: We don’t have tests for that, but that’s a great idea. Where we really like in the HMD area, so we certainly will be looking into that. Unnaturally this whole VR and then AR as we see being even bigger, that’s requiring many many years of product test development to enable the industry like the important decisions.

VRguy: You’d think that if I have a very high performance HMD but then lousy optics in front of it, then that significantly dampens the user experience.

Tero: At the moment it’s subjective tests or some high speed cameras, but there’s got to be something more scientific, and we certainly will be looking into that.

VRguy: So when you did the VR benchmarks, what was different about VR relative to the graphics and other PC tests that you’ve done before? What did you find most difficult?

Tero: VRScore for PC is developed, first and foremost for DX12 API, and what this means is when we have a totally new graphics API which is more difficult to program than any of the previous ones because it’s more of a thin API, that’s one thing. Then we have the new graphics cards, new GPUs with new drivers and we have HMDs with their set of drivers, their SDKS. They certainly introduced yet another new layer of complexity and sort of matrix of permutations in the things to test, and ask which things work, which it’s other, and which not. It’s been over the last three months we’ve been just big fixing and sort of performance optimizing.

VRguy: I understand, so it’s not so much the specific to VR, but more about just new programming environment.

Tero: In VR it’s not your traditional graphics performance test where you just measure FPS and that’s it, because in VR it’s all about maintaining the presence, the experience. So you need to account for the dropped frames or duplicate frames and things like this, also in sort of the scoring part, and so that’s been a big part of new development.

VRguy: Excellent. So, if you could control the development plan of Oculus and Valve and Sensics and Microsoft and other leaders in this market, what would you have us do to improve the user experience? What would you have us focus on?

Tero: Of course, as you mentioned the updates, and it’s about the screen pixel density and the stability of the frame rate. So, most likely those are the biggest things to focus on.

VRguy: Very good. So, Tero how could people connect with you online to learn more about what you’re doing.

Tero: www.basemark.com, or you can just send me an email my first name is [e-mail]. My pleasure.

VRguy: Tero, thank you so much for coming

The post VRguy podcast Episode 21: Tero Sarkkinen, CEO of Basmark appeared first on Sensics.