AnsweRED Podcast Episode 27 — From Feedback to Fun: UX Research in Game Development

Paweł B: Hello and welcome to the AnsweRED Podcast. My name is Paweł and I am joined by Paweł Mielniczuk, the Art Director for Project Hadar.

Paweł M: Hey everyone. And today we'll dive into a fascinating world of user experience research with our guests, Joshua Marchand, a User Experience Research Lead, and Dawid Zalewski, Senior User Experience Researcher and Research Operations Coordinator.

Paweł B: Wow, that was a mouthful. I can't wait for this episode. So let's meet our guests. All right, Josh, Dawid, welcome to the podcast. Officially. We like to start these off with you guys introducing yourselves so people listening to the podcast can associate the voice with the name. So, Dawid, do you want to start?

Dawid: Yeah, of course. Pleasure to be here. My name is Dawid. I'm a Senior User Researcher and Research Operations Coordinator here at CDPR.

Paweł B: That's a mouthful.

Dawid: Yes, yes it is. I'm responsible for running playtests and other studies with our players, bringing them to the table and asking them what they think about our titles, about our games and development.

Paweł B: Really cool.

Josh: And, hey there. My name is Josh Marchand, and I'm the UX Research Lead here. And it's a lot shorter than yours, which is nice. So my job entails basically making sure that the way we use UX research at the company is going to get us things that make us confident on how players are going to experience the game when it comes out. So it's a little bit of strategy, a little bit of planning, little bit of working with Dawid and stuff like that to work on it.

Paweł B: Awesome. So let's start with the basics. So what is UX research? And when did we start this initiative? And tell me a little bit about the origin story.

Josh: Yeah. So UX research is—it's a good thing we have a long time to talk about it. UX research is essentially the use of things like behavior psychology and cognitive psychology to understand how players will interact with the games before we actually put the games out in the real world, right? So if you ever played a game before, you know, if we go back to like the 90s and you're playing video games, if you remember playing one and you're pissed off because you don't know what you're supposed to do, you go to gameFAQs, you go to YouTube, you go somewhere like that to figure out what you were supposed to do.

Paweł M: There's no YouTube.

Josh: That's true. There's no YouTube.

Paweł B: That time I would go to a big grocery store and just go and get like the game guide with paper. Oh, I remember I used to love to do that.

Paweł M: It was on forums.

Josh: And so yeah, and forums and stuff.

Dawid: And if it's not out, then you're waiting.

Josh: And then you just don't know what to do, right? But when you open up that game guide and you look and you figure out what you were supposed to do, do you ever remember rolling your eyes and going, how the hell did they think I was going to know to do that? Right. That's what we want to help you guys and the developers not have that experience happen for players. We want them to feel the experience the way you want them to have the experience. So we do playtesting and other forms of studies to understand how players are going to interact with it. The fun way we tell people our job is, is we get to watch people play video games for a living.

Dawid: That's how I usually describe it. I mean, we obviously were doing user research in the past, mostly with external partners, with vendors, but the internal initiative and creating the team, it was, I believe, September 2022. So a year before the release of Phantom Liberty.

Paweł M: Okay, so that's a relatively new department in our company. So what were the first challenges actually of introducing this kind of new domain into the company?

Dawid: There were actually quite a few because like, if you think about it, that, you know, previously we were mostly outsourcing, working with outsource is pretty specific. You usually tend to work until the very end until the thing is final. It's very often one shot. It takes a lot of effort, a lot of organization to pull it off. So you don't really get the chance to test regularly. And so creating the team at CDPR introduced a new way of developing games, a way where you can test regularly, where you can work iteratively. A way where you don't need to wait for the thing to be final. If you have a paper prototype or you have a concept, you have an idea, you can come to us and we can start working on it and help you evaluate it and help you improve it. And so obviously, there are people who are eager to jump on the train and start working with us, and they saw us as a tool for them as a support role in an RPG, helping them make their life easier. But there were people who were reluctant, either because they weren't experienced with this research before or because, and this is a common belief that, you know, because it will take away the creative freedom. And we are an ambitious company, we are a creative company, and we want to make games the way we do games, the CDPR way. So there were people who were concerned and who were afraid that, you know, now they are going to lose that independence and they will not be able to, you know, express themselves artistically how they would want to. And so we had to work with those people carefully, understand why they are reluctant, why they are hesitant about it. Help them explain that, hey, if you want to create a game that is difficult, if you want your players to suffer, that's exactly why we are here. We will help you achieve it. We will not tell you that you need to design a game differently. We will try to help you achieve the design intent that you have. And slowly over time, we managed to build trust and we managed to build confidence in our department. Also the people who were eager to work with us, they were great to help us achieve that because, you know, it always resonates differently when I tell designers what we do versus when other designers talk with them about the experience they have with us.

Josh: I would say in terms of challenges with that as well, is UX within games can be quite messy and that's because like when we look at the UX of a phone app or a website or things like that, the goal of that app is to efficiently let the user do what it needs to do with the app. If it's a banking app you're supposed to look at what your account is and you know, you cry when you look at your balance and things like that. But with a game, the use of a game isn't about efficiency. It's about the experience that you're having. And that's even more true at CDPR than other companies as well because of the storytelling things. If you look at Tetris, Tetris isn't a game that is about the emotional connection to a main character and things like that. And so UX can be quite messy depending what type of game you're making and like what Dawid said is the experience of using user research with that process. And so something that we find is whenever you move to a new company trying to do user research, you can't just say, well, I did it this way at this company. And so it's going to work this way at this company. You have to understand and figure out almost from ground zero, how you're going to do it here to address the problems and needs of the company and so on top of everything Dawid said it's also playing that game of creating a footing, creating a grounding so that we know what does user research look like here at CDPR so that you guys know that when we're doing things the right way and we know we're doing things the right way?

Paweł B: So how did you guys define good UX research? What is good UX research? So how, if you could like talk about the process, maybe.

Dawid: So when you think about UX research and applied research in general, I would say there are two main approaches. There is data-driven approach and there is data-informed approach. And so data-driven for those who don't know is the approach where data basically dictates how you develop things. If data says X, then you do X. Data informed on the other hand is there to help you make the decisions. But ultimately you are the decision maker. You still maintain the creative freedom. And if you feel that this is the moment where we should take the leap of faith and trust our guts and do the thing the way we want to do it, then we do it this way. So I feel it was a no-brainer when I looked at CDPR and I had conversations with many of our talented designers that they are passionate, that they are experienced, they are pros in what they do. We definitely wouldn't want to limit them in any way. It should be a tool that helps them in their day-to-day work, not something that micromanages them and tells them, okay, so you need to do this and this and this and that.

Josh: So the way we would, to kind of complement what Dawid said is, we know that we've done a good job with UX research, we know what quality UX research looks like here at CDPR if our partners that are developing the games know how players are going to interact with the experience, if they we have armed them with the knowledge to know if players are going to experience you know, this very key part of the game, the way that they want it, we've succeeded. Like it's not about... for anyone listening that that maybe does UX research or UX in other software and things like that. It's not about making sure that the action is performed within a certain amount of time. It's about the feeling and it's about the experience that comes from performing the action. And so we don't try to judge success by users are able to do this within five seconds of being presented the option, because that's not really, it sucks the soul out of something that you can't really have a good video game without that soul, right?

Paweł B: Totally, totally. Well said. So if you could talk about maybe some implementations of things that kind of, you know, your team worked on and how it kind of informed development. Can we, we could probably take Phantom Liberty as an example, right? Because that was already where the team kind of started, like before the launch.

Dawid: Phantom Liberty, but I think the Ultimate Edition of Cyberpunk on Nintendo Switch two would be even better example, because it was a very interesting thing to test. And for those who haven't had the chance to play it, in the Ultimate Edition, we introduced motion controls so players could control their characters with the movesets that we defined. And obviously they are specific. They have to be in order to players be able to effectively use them during the game. But they for some were not really bringing that power fantasy to realization. So we had one of the players who started playing, slowly he was following the movesets that we have established. But then the longer he played, the more he got into it. He started smashing the Joy-Con, really getting into it. And because we didn't think about it before, it was unreliable and unresponsive half of the time. He didn't mind. He didn't care. He still enjoyed it. He didn't even realize that it's not working. He managed to kill the enemies. He managed to move through the level. So it was fine. But after that, and after seeing it, we had a conversation with the designer that, hey, he really loved playing this way. He really got into the game. He was so immersed. What can we do about it? Like, can we try to make it possible for players to play this way, not as an official way? We are not tutorializing that you can play this way, but first of all, it doesn't really harm the experience of other players. It doesn't go against the vision that we have had. If anything, it only strengthens it because one of the ideas we've had for the Ultimate Edition on Switch 2, and there was a reason we say it's the most cyberpunk way to play Cyberpunk was for the experience to be as immersive as possible. And so, since it was not difficult to implement, it was easy win. We decided to go for it. We decided to implement it. And then we tested it on the next iteration, and we had more players and more players playing like that and them enjoying it even more than before. It was a really, really good experience and it was really amazing to see that the fruits of your labor coming to fruition and being able to see how even a small change, even a single player, because it started with one player acting this way can influence the game and the direction.

Josh: I really like that example too, because something that Dawid kind of snuck in there, which was the iterative part of it, right? It wasn't that we tested it once and then we made these big decisions based upon one thing. It allowed us to have food for thought and the developers took it and tweaked something else. And then we tested it again to see how it works. And that's where we really shine. It's not that we come from our tower on high and give you a Bible to say this is how you need to make the game. We're testing. We're trying things out with you guys. We want to test with you frequently and often and to understand, are we going in the right direction? Like it's not did we do it the right way? It's are we going the right way for it?

Dawid: And it's always a great experience when, after several iteration, you finally reach the moment when it all fits together, it all starts working and you have that confirmation and that positive reinforcement that we were going in the right direction all the way. Thanks to the improvements in iteration, we managed to get the experience to the state that we wanted it to get. And I don't know if you're familiar with the anecdote of how Ewan McGregor and Hayden Christensen were making the lightsaber noises on set. So we had a player who was exactly like that. So, it was one of the later playtests that we have conducted. We invited a player to test the melee build and he started slowly.

Paweł M: Thermal katana.

Dawid: Yes, yes, exactly. So, at first it was a little bit shy. Of course we are there. We are observing, we are taking notes. But after a while he got so into it that he started making the katana noises himself. He was slashing the enemies and going like [swinging noises] it was like, it's a great experience because very often user research is associated with the bearer of bad news that, oh, this doesn't work, that doesn't work. We need to fix it. But it goes the other way too. And it can be extremely motivating for the team to see that what they have created is enjoyed by players, especially before the release. To have that confirmation that we are going in the right direction, we should keep going. We don't need to wait six months, 12 months, 18 months to learn whether it's good or not. We can know right now.

Paweł M: Yeah, I think, but also it's a great point that, we as the developers, when we are designing some systems, we sometimes get into the trap of getting used to play it in a very specific way. Right? Also the quality assurance team, they also might get into trouble because we're working on the product all the time from the beginning, for years actually. So you don't have the distance in the perspective. So then inviting players who will show us how they intend to play it just, you know, completely discarding our guidelines. That's a great thing, right? Because then when we release the game, you know, millions of players actually at the same time start playing it and you cannot foresee it actually.

Josh: That's a great point. You want to be a user researcher? No, that's a fantastic point. And that's something we work constantly with to help people understand is, one of the things that in game development that we love saying is we make games for ourselves. Like we make games that we would want to play, right? And that's a good thing. And that's what we want. But also, you're not going to buy the game a million times. Like I'm not making the game for you. Right? Like we're trying to make the game in a way that you can explain. You can make a game that you want to play, but someone can play a game that they'd want to play as well. And so, we encourage developers to play the game on their own. Like the more you play the game, the better because it allows you to kind of see the game in motion. But we would never say that that replaces bringing, you know, like a player coming and sitting down and talking and us cracking into why they're experiencing the game the way they are.

Paweł M: Also another aspect is that I remember Phantom Liberty, I was playtesting internally the game also with game director just sitting, we had this internal test of the game with the whole development team was trying it. And you know, I'm playing games, but I'm not a hardcore player. I'm, I think I'm a more casual player, you know, not super skilled.

Paweł B: No one's judging.

Paweł M: And very often game directors and game designers, they're very skilled players. And I gave this feedback, guys it's great. But I keep dying all the time. Just cannot beat this part. Oh really? And they were ultimately surprised right. But it's also great. We are not marketing our games targeted towards the hardcore players. So it's great to make it accessible, right? So if you want to play the hardcore way, you know, get experience, learn, there is a way to do it. But if you want to play on easy mode in the narrative mode, it should be also accessible for you.

Josh: And also with the example that you just gave, if you ever had a job that wasn't in game development or things before, you always try to find quick ways to do things efficiently. Like you don't want to do things the inefficient way, right? So if you ever worked at a, you know, being a cashier or something like that, you found the fastest way to press those buttons to get someone out because you wanted to be done with it, right? I mean, that's true when we're playing our own games as well, is when you're trying to get from point A to point B in the game and you need to do that 200 times, or you need to do that, in between meetings and in between the planning sessions and things like that. Especially when you're very busy, you're going to do it in the fastest way possible, not necessarily the way that players would play the game. So you're accidentally, even if you're not consciously thinking about it, you're finding ways to play the game the way that it wasn't intended to be experienced. And so we want to help, like open eyes, give the perspective of the game being experienced the way that you claim it should be experienced, right?

Dawid: And, you know, as developers, we should be playing our own games. We should be familiar with them testing them inside out, but we will never replace players. I really do feel that it's important. What do we think about the games we create as employees, but not necessarily what do we think about them as players? Bringing that perspective to the table is extremely important, and I think we can all agree when I say that it directly ties with our company values. Always remember about gamers—what's the better way of remembering about gamers and bringing them to the table and making them part of the development process?

Paweł B: I really like that. I also like that what kind of mentioned is like, we have the possibility to test things very early on because from my perspective, whenever we are working on communication for a specific game update, whatever, we play it, but we play it at the latest stage. When it's super ready, it's polished and we kind of can, you know, think about what are the key features that we should highlight, how we should build the pillars around stuff and stuff like that. But, you know, it's good to actually go into the process very early and test things which are very early in development. And this is what you're also doing, which I think is super cool because I think people might have this notion that, oh, so you do like, user experience research based on a title, which is already complete, which is totally not the case. We sometimes take just snippets and we just take those like specific things and we test them, and then we kind of see what the feedback is, which I think is like super, super cool. And it lets us, I feel like it lets us kind of be earlier hands on with it and then seeing the evolution, which also informs the way how we do communications, how we change things within the game and stuff like that.

Dawid: I mean, to be honest, sometimes we don't even need the game to test the game. We can start as early as paper prototypes, and it is always also interesting for for players when they come into the lab and, you know, they expect to play a video game and then they see paper cards on the table and we tell them that, hey, this is the game we'll be playtesting today, and it's even better when they are actually enjoying it at the end of the session. So yeah, earlier we can start the better for the project because there is a psychological effect called sunk-cost fallacy. The more resources, the more time and effort you invest in designing something, the more attached you grow into it. And then it's difficult to change it. Or sometimes we are restricted by the timelines and we can't change it if we want it to. If we start early, if we test things that are in a scrappy way that are not final, that are very early stages, we can easily throw it to the bin and try all over again if we are not happy with the results, if we're feeling that we are not achieving what we wanted to achieve with them.

Paweł M: Maybe let's go into the details. So how do you actually design the UX study?

Josh: Ooh. So, something that researchers say a lot is, it depends. First we have to identify a couple of things. The first is, why are we doing a study? What are we doing the study on? And how is this going to help the people asking for this study, right? So, Dawid, a little bit earlier talked about applied research, right? If we get very general, there's kind of two camps of research. There's the things that are more traditional academic research. And then you have like applied research, right? Academic research is more knowledge for the sake of knowledge and learning how things work. It's really good. It's really awesome. But if I come to you with that type of information, you're not really going to be able to do anything with it, right? So we have to make sure we're in the camp of applied research where the research is usable for a problem you're trying to solve now. So if you came to us today and said, I want to know if if jumping is fun in this game. Well, to us that's not great because that's more of an academic research camp, right? You just want to know something, right? Well, what are you going to do with that? Is it just to make you feel better? Well, what happens if people don't find it fun? That's not going to make you feel better. It's going to make you feel worse. And then we're not really doing anything at that point, right? So it's drilling down. What are you going to do with it? Are you going to fix jumping? Is this about an emotional resonance? Is this about people being able to do the thing? So we identify that in a very meticulous way that allows us to then actually design the study. I know that was very long winded, but we spend a lot of time sitting down with our partners, understanding that part so that we can do the actual design.

Paweł M: And how gradual you can go actually? Are you testing individual features or there's like the whole experience or you—

Josh: I can talk about my pyramid. So a big thing that we try to help because this goes towards what Paweł what you were saying earlier is that a lot of people expect that we come in at the end after everything is done, and then we test. We actually don't like to do it that way because of a lot of things. So we actually like more of a pyramid or a triangle where the base is those individual features. So we want to test individual features first, then move into studies that focus on like systems, which are interconnecting features, and then finally going up to what we call more high-level UX, which is more do they have fun with it? So when we're looking at features, we're probably not even going to be looking at, are they having fun with it? It's, can they do the thing? And if not, what's causing them not to do the thing right? So the reason we do that is if we start with the end and we ask the question, do people like this game? Right? Well, if someone is having a controller problem, like a button input problem and they're not able to jump well, if you ask them, do you like the game? They're going to tell you no, but that doesn't mean you necessarily know why they don't like the game. You just know they don't like the game. So we like to start with individual features so that we can identify that. So that when we ask questions like do you enjoy the game, we already know if people are having problems with that. So it's not affecting our ability to know the bigger picture of things, even if we're still working on them.

Paweł M: I think it's also sometimes might be hard for the players to tell you exactly why they don't like it, because you have to actually do something which is very hard, like measuring fun, you know.

Josh: We actually call that the F word, like fun is the F word to us because everyone's sense of fun... I mean, if we went around the room, everyone has different games that they like, I actually love fighting games. I have a blast with fighting games. Dawid knows I can sit there and play them for hours, but that doesn't mean everyone has fun with them, right? Like fighting games are often frustrating and people don't really want to sit there and play them for hours. When I was younger, I used to go to an arcade from seven in the evening to seven in the morning and just play for 12 hours straight because I just loved them. So my definition of fun is severely warped compared to someone else's definition of fun.

Dawid: I don't like fighting games.

Paweł B: Same, same. I'm more of a cozy gamer or RPG.

Josh: And so if I asked you what fun meant, you're going to say something fundamentally different than I'm going to say. So we don't want to even get into that engagement. So if a partner comes to us and says, I want to know if this is fun, what do you mean by fun? Like, so that's where design intent comes in. Really important is understanding. You have a sense of what fun is supposed to be as a designer. We don't want to try to give you a thumbs up or thumbs down on your ability to be a designer. We want to give you a thumbs up, thumbs down if the player can experience what you wanted. Because we come with the belief that if players experience what you want them to experience, they will have fun. It's a designer's goal to create the fun. We're just there to help get the fun to them, right.

Dawid: Basically, if you're coming to us to test the combat system and you tell us that in the combat system that you envisioned, players should be jumping all the time. We are going to invite players. We are going to see if they are jumping. If they are not, then we will try to understand why. We'll try to understand what we need to change in the system together with you to make them jump. You will introduce some changes and then the next cycle and we're trying as long as we'll manage to achieve it, or as long as you as a designer will tell me that, okay, we are changing the direction. I no longer feel that this is where we should be going to.

Josh: And something that you also said earlier is that's really important for us, is understanding why people act the way they do and things like that, which is why our job is kind of important, you know, job security and stuff like that, is everyone—and we encourage everyone to like with your coworkers, to sit down and do kind of unofficial mini playtests, right? Like sit down with your coworker and say, what do you think about this? Um, because you can get good things, but our background isn't just in, in that. Our job isn't about running the research. Like I can give you a moderation guide and you guys could run this study and it would be fine. It would be fine. For the most part, where we really come in is cracking into the patterns of why someone reacts to something the way they do. And is that something we should be concerned about when the game comes out. And so you can rely on us to help, you know, if this is something we should care about or if this is something we can just go, I see it now, but this isn't going to be a problem in the future.

Dawid: And as you said yourself, people are not always or most of the times aware why they enjoy or don't enjoy something. They have ideas, they can give us hints, but it's very common that, you know, players are trying to give us design suggestions that, hey, I don't like this, so you should fix that. But then when we dissect the problem, we often come up with, we get to understand that the problem is somewhere else. That we shouldn't change the thing that they dislike. We should change something else because changing something else will actually impact the experience much, much more than what they initially suggested.

Josh: We actually see that happen quite a bit. Video games are messy, right? Like, it's the butterfly effect. Yeah, it's the butterfly effect all the way down. If you change one thing, you don't know what that's going to do to the complex experience that people are having within the game.

Paweł B: It's also taking multiple pieces and trying to stick them together and see if they're cohesive and working together, which is crazy.

Paweł M: Okay, so you got this inquiry for let's try something, let's do a playtest about some feature. Then you do the research on the developer to understand what he really wants to try and what's the intent he was looking for. Then you gather the playtesters, you conveyed the research. They either play it, read it or play it, depending on what stage we are and what happens next? How do you now collect the data and what, what's coming back to the developer?

Josh: So it depends.

Dawid: Yeah. So what's coming back to the developer most of the times is a report outlining what we have discovered. So based on the study protocol that we have established and based on, what we tested, we're trying to explain what happened, but most importantly, why it happened. So not only the players had issues with X, Y, Z, they had issues because of this and this and that. And that's because we don't want to deliver insights. We want to deliver actionable insights, something that you can take and something that will tell you what you should change, fix, something that will guide you, something that will help you with the decision making. If you're not sure which direction to go or if you're not sure whether you know what you're trying to do is the right thing. And then the report we are delivering should help you answer that question. It will not answer the question for you, but it should be one of the pieces that you can take into your decision making process to make it easier.

Josh: Go ahead.

Paweł M: No, just one step before, to make the report, that's for sure. There's a survey. What else?

Josh: So once again, it depends. What we have to do. So before we have participants come in we design the study itself. Josh: So think about it. You know, we do science, right? So like we are trying to make sure because for us, it's not about, you know, five guys telling you what they like and don't like. It's about the ability to extrapolate that into the audience. And so we need to make sure that we are objectively confident that the stuff that we are giving to you is going to be as close to fact as we can get. We're not going to guarantee, people are people and things like that, but we try to do that. And so we want to ensure that we understand what success looks like, what failure looks like for the things that we're testing. So if we're, you know, we keep talking about jumping, right? How many times can a player fall into a pit before it's considered that we have failed in designing the jump feature, right?

Paweł M: Jump in Cyberpunk.

Josh: Exactly, exactly. And so we want to understand that beforehand. And so that way we can design data collection methods that allow us to collect the important stuff and not worry about the excess data. Because when you have participants playing a video game, there's so much data going on. There's what they do, what they say, what they look at, what they do. So if you're looking at a game like Cyberpunk and you're going to have them play for three hours, think about all the little things they do in the game. Those are all data sets, and we don't want to collect all of that, because then that's going to take us four years to look at all that stuff, right? So we want to focus on specific things. Sometimes that means survey. So if we're looking at what people think or what people like, we want to get into their heads a little bit, surveys are a great way to do that. We also record the the gameplay so that we can look back at it. We create things that we call data sets, which are behavioral things that we can track. So did the player jump at minute 57 of the thing? Check or X, right? And so that allows us to have multiple ways of trying to understand what that experience is in an objective way, rather than just what the participant said or thought.

Dawid: I would say the tools are only means to an end and that they are there to serve the objective. So it's easy to get into the conversation about, you know, oh, there's eye tracking, it's fancy. We can see where players are looking at. Or maybe think about protocol that we ask them to explain live their thought process so we can get an insight to it. But it's also very easy to pick the wrong method. And then if you pick the wrong method, you're getting wrong answers. So if you look at eye tracking, for example, the mere fact that somebody is looking physically at this particular side of the screen first doesn't mean that they are consciously registering it. And second, it doesn't mean that they don't see anything else because we have central vision and we have peripheral vision. So even though we don't really focus on things on the sides, we still register them. And so when we started designing a study, when we started thinking about how to measure the thing we want to measure, we need to take a lot of things into account. Like, okay, what if I ask this question before that question? Because this can affect how they are going to answer, because now I'm revealing some information or I'm revealing the intent I have behind the study, or what if they play this part before that part, and all those things have to be taken into account before we even start the study. That's why when we were going to the report preparation, we already more or less know how it's going to be structured, what we are going to convey. I would say that designing the study, designing it right, is the most difficult part of our job because if we made a mistake, then—

Josh: You can't go back.

Dawid: You can't go back and then sometimes it may be difficult to realize that when you get the data, you may deliver misleading results. And that's, I would say, the biggest part and the biggest responsibility we have as a user researcher running the session. It's not difficult. As Josh said, we can give you a guide and you can assist in the next one if you want. But actually being able to, you know, connect the dots and foresee the problems that may happen and mitigate them before they happen, that's the most difficult part.

Josh: I would say quite kind of with what Dawid's saying is, I would say that we spend most of our time making sure and asking the question, are we going to be confident with what we get from this? If we can't be confident with it, we don't do it, and that might sound somewhat counterintuitive that you might come to us and say, hey, I want to do this study and we'll say no, because if we feel that there's even a slight chance it's going to put you on the wrong path, we wouldn't be doing our jobs if we let you go down that path. We don't want you coming back to us in, you know, two years after the games' out, going "That didn't go the way I wanted it to go." Like, we need to kind of keep the guardrails up to make sure we're confident in it. And so that's the reason, like Dawid said, we're very particular at that part. Now we can get into the fun part. Like, you know, afterwards, right?

Dawid: But it is kind of counterintuitive that, coming from researchers, that we also believe that sometimes educated guess and years of experience we have is better than running study if we are not 100% sure that we are able to answer the questions that you're asking.

Paweł B: Plus also not running into a place where you're testing things just because you think, oh, only the science will work, right? It's still kind of like you have to sometimes put science aside and kind of think about, you know, maybe the developer is right. Maybe if they want to do it this way and it feels like it's working, it's working. We don't need to like really dive deep into it. I think the biggest problem is when developers kind of think like, oh, is this going to work? Is this not going to work? Are we going in the right direction? Can we maybe test if we're actually, if we should really put more emphasis on this, if we should put more work into this.

Dawid: Sometimes we just need to take a leap of faith and follow our instincts.

Paweł B: And sometimes that's where the magic happens.

Dawid: Exactly, exactly. And that's another thing that, very often when we're delivering results, we tell designers to be very careful and to be very aware of the consequences and maybe don't jump to conclusions straight away, but give it a thought. Think about, will the change that I'm planning to implement based on these results be aligned with the creative vision I have? It's easy to implement solution, but it's more difficult to implement one that will both address the issues that we have discovered, but also serve the overall vision that we have for the game.

Josh: When we look at developers getting more familiar with working with user research, one of the first levels you're at is kind of rejection. You don't want to work with a user researcher because it's sucking the soul out, it's science, and things like that. You run into the fear that they're going to tell me how to make the game. And so you don't want to do it. It's not fun to be there, but it's fine. We're used to it. Like we work with people to understand why that's not true. It's a natural part of the process. The thing that becomes somewhat dangerous for user researchers is actually the next level up, which is where you get familiar with it. You like us, you kind of, you know, maybe you have fun with us and you want to do more of it. And then we give you a report. And because you've built up that repertoire, your first instinct is to just do everything that we said in the report. That's more dangerous because at least in the below one, we're having to prove ourselves. If we just go in there and you just do everything we say, that's not our job. Like Dawid said earlier, our job is to be a food for thought point of information for you, not your say-all Bible on how to make the game, right? And so we want to get out of that phase of our relationship as fast as possible to where you actually sit there and ask us, well, why is that the answer? Like, why should we do that? Challenge it a little bit. So once we get past that point, that's when we get into the really fruitful conversations, right? So we do not want a situation where we give you a report and you just go, cool. And then you do everything that we told you without even asking why. And that's, Dawid and I both have seen that in our career where people just take it and be like, okay, I'm going to do this. And it's like, hold on.

Paweł M: Great.

Paweł B: I love that you mentioned science because I remember when I did my first playtest, I felt like I was being watched from every angle—

Paweł M: It is like a laboratory.

Paweł B: And I was—Yeah. And I was thinking about it like, back in the day, if you grew up playing games, like I was playing on NES, I don't think anybody would think that, we're going to have like researchers sitting down, like doing like these deep studies into what players do and stuff like that. It's incredible. Like you wouldn't think that games and science like this is actually a field that is going to come together, but it actually did.

Josh: Well, I could go on a tangent. So there's an academic field called ludology, which is actually the study of games and how games interact with people. There are fascinating books like Homo Ludens, which kind of has the assertation that people at its core learn through play. And so that's the reason why we like video games because it's a safe way of learning. And we get the reward from solving a puzzle without the danger of the world around us. And so there's a lot of gold to have in kind of mixing science and games together. We just make sure very carefully that we don't put too much science where it's no longer the soulful experience that we want, right? Like we don't want science to overrule that heart that games have. Right? But there's a lot of fun to have with science and games.

Paweł M: What we can observe in recent years is gamification of life, you know? Applications or companies outside of gaming that are actually using this gamification. You know, this core loop of you're getting some, you have some challenge, you beat the challenge, you get the prize, you learn something and then you apply it to next challenge. You can even see it in banking apps sometimes. You know, you're gathering—

Josh: The level up stuff.

Dawid: For a very long time, and video games are just a new incarnation of something that we have known for hundreds and hundreds of years.

Paweł M: Yes. Um, so maybe, what are the fields of research inside? Because obviously you are focusing on gameplay, right? This experience of how you play the game, how fluid it is, etc. But what are the areas? Is it—

Josh: It depends. Oh, so gameplay is the thing that's the most natural because UX—actually, I would say that's not even true because a lot of the times when we're working with developers who maybe came from different industries, they're used to user research in their industries. Well, if you're doing UX on a phone app or something like that, I would say probably 98, I'm pulling a number out of my ass, but 98% of the UX involves UI, right? And so because of that, the most common misconception is that actually UX equals testing the UI on games, right? And so that's like the first one that we're like, yeah, we can test that, but that's definitely not the only thing we do. Now, when you're from a gameplay perspective, you think about gameplay. I think the easiest answer for this would be that we want to test anything. We want to work with anyone that is putting something in front of a player that is going to be experienced, right? Within the context of the game, right? So that's art, that's music, that's anything that is going to affect the way that the player is experiencing in a way that we wanted it to be a catered or intended experience, right? So everyone.

Dawid: Yeah, but I think it's natural that the gameplay is the first one that comes to mind, because it's one of the easiest for us to approach with enough objectivity to give results that, hey, you should fix that. You should change this. It's easy to discover problems that players shouldn't be encountering. Like if you can't find a specific tab in the menu, then obviously that's not something that we want. If during the combat you struggle with pressing buttons, that's also not the friction that we want to implement. There should be friction in games. Friction is needed, but with gameplay, we can set objective and behavioral metrics that allow us to track and be like, deliver reliable insights. If we look, for example, at narrative and story, we obviously would love to test it. We do test it, but it is much harder. It is much more subjective, and we may easily run into a problem that one player says X, the other player says Y. They both have some valid points, but in the end, it is us who need to make the choice on what we're going to do with that, if anything. It's difficult to find objective metrics when evaluating story because—

Paweł M: Emotions for example.

Dawid: Exactly, exactly. Like, experience is what's happening in our brains. That's why it's so difficult to measure. And what we are trying to do with user research, we're trying to apply as much objectivity as possible. It's easier with some areas than the others. UI, gameplay, not so much with narrative, with music.

Josh: I would say a lot of that difficulty also comes from the fact that with gameplay and UI, you are player centric in the sense that it is evaluated by what a player can do. It requires a different way of thinking significantly for narrative and art and sound to have to look at a player centric perspective of it rather than the creative output. Like, it's not the inherent way we do creative in a lot of those fields, which means if we're going to try to test it, it means we're asking them to do a lot. Like they're having to change the way they're doing it. Because if I come to, someone that's creating a piece of music for one of our games and I say at this point, what is the player supposed to feel at this point? Well, they're going to give me an answer that is not necessarily like Dawid said, testable, right? And I'm going to have to walk through with them and say, this happens at this point in the game, the player is doing this. What does that mean they're supposed to be feeling? What are they supposed to react to it? So that we can then reverse engineer metrics from a behavioral perspective, right? Like how do they perceive it?

Paweł M: They're also missing the whole context of what happened before, because I'm sure you're not—

Josh: It's very hard to separate it all out, right?

Dawid: Yeah. And then it's also like, how should you treat your results based on what we are testing? When we can introduce some objectivity, like behavioral metrics, and very often it's straightforward struggle with X, we fix X. If players don't like the story, then it's food for thought. It's something that you should take with a grain of salt. It is important we do value what our players have to say, but then ultimately we need to make the decision.

Paweł M: Especially that sometimes we want players to be like, we want some designs to be polarizing, stories to be polarizing. Like, for example, I guess if we'll be testing Cyberpunk, the narrative I think will introduce a happy ending at the very end, but there's no happy ending. Nobody like that.

Josh: And that's why design intent is so important, right? It's not necessarily about players being happy or sad. It's are they sad when you wanted them to be sad? Are they happy when you wanted them to be happy?

Dawid: But that's actually what happened with the Phantom Liberty ending when we were playtesting it and players were sad and some were even crying in the lab. Uh, and—

Paweł B: Not surprised. I had the same thing at home.

Dawid: No, no. Uh, and so like, some of them were disappointed because they expected happy ending in the end. They expected V to live happily ever after, But that's not what we wanted to achieve. That's not what we wanted our players to feel. And so the fact that they aren't particularly happy that we went for the for the sad ending again, it's not really a problem. That's what we wanted to achieve. We managed to achieve that.

Paweł M: A design decision.

Dawid: That's the design decision. So there's nothing to be changed here. It's only a success on our side, even if they are not fully happy at first.

Josh: And that's why it's so important that we look at it in terms of how familiar you are with working with UX research, right? Some of those questions are a lot more complicated to answer, and it requires a nuance. But like Dawid was saying, with behavioral based stuff like gameplay and UI, it's show and tell. I can show you that players aren't doing it the right way, right? It's a lot harder to tell them that they aren't emotionally resonating with an ending. If you don't have that level of understanding of how the work we do is. And that takes a long time to understand because as I'm sure you guys are understanding, we're dropping a lot of terms and a lot of things—it's a lot to kind of juggle the different spheres that we have to do to be able to do this work in a good way, in an efficient.

Dawid: How would you even measure if they're sad or happy for us to call it a success?

Josh: Yeah. It's like is it a scale? Which we hate scales.

Paweł M: I guess it's even for playtesters to answer some questions, it might be sometimes uncomfortable or maybe, you know, I'm sad because of some events in my past that got recalled after playing the game. It's, you know.

Josh: We try to mitigate that stuff as much as we can. But once again, humans are very complicated things. I can't guarantee. Like we do our best to mitigate things like, if you're playing a video game and there's a scene about spousal abuse and you just got out of a bad relationship with your spouse—we can't account for that. And so we have to on the fly try to figure out when those types of things are affecting our results, because we don't want to accidentally take that one experience and then extrapolate and say, yeah, everyone's going to have this problem, or everyone hates this because of something we didn't really take into account. I think that's part of the fun though, right? Like it's part of the fun for us is nothing happens the same way twice. When it comes to designing a study and running a study and trying to address problems.

Paweł B: Correct me if I'm wrong because I think that UX research can actually enhance the experience in games that we're kind of going for, which is always like these hard choices and consequences. So kind of having branching quests with multiple options and seeing what players do and which direction they go. Of course, there's always a player that goes a totally different route and does something totally different, but then I feel like, with enough data points coming in, you're able to kind of see, oh, players do this or sometimes they do this and then the reaction is that. So it's like a nice balancing act, but also enhances the experience. And then the quest designer is like, oh, maybe I should adjust a little bit this way. So maybe more players will do this. Or maybe let's just leave it totally open and let players do whatever they want, right?

Dawid: I mean, I would say that it's easier to get enough data points after the game is actually released with the support of our friends from gaming analytics team. We can get telemetry data and see large scale how many players chose this particular path, what they do. We can follow up with our qualitative studies to understand why. For example, we have two paths, two branches, and one is clearly favored. Then we can run studies to learn and understand why it happens. So in our future games, we can try to avoid that problem. We can try to make sure that both paths are equally compelling. Of course, if that's the goal. Yeah. But yeah, I mean, being able to observe the choices players make is definitely half of the fun. Especially that as researchers, we see the segments of the game hundreds of times. And you know, especially if we're running larger playtests and we have ten players at the same time, I see ten gameplays simultaneously. So it's sort of like, sometimes I do feel like Neo in The Matrix where he got the kung fu uploaded to his brain. I get the game uploaded to my brain just by seeing everything that's happening. So that's definitely part of the fun. And it's definitely important to see it as soon as possible. What are the natural tendencies? Because then we can use it to our advantage. If we see some players are doing X, then we can give them more space to do X. Like with the example I gave with the more expressive way of playing Cyberpunk on Nintendo Switch. So that was the opportunity that we discovered through playtest, because each and every player played differently, and it showed us many different directions they all could go.

Josh: So you guys are tapping into like a big thing, which is like quantitative data versus qualitative data, right? Quantitative data is what you were talking about, Paweł, where it's like tracking how many times people do something. It's really good at telling you what people are doing. What it kind of falls flat on doing is why are people doing it? So you understand that 50% of people are taking this path. But a developer might say, okay, but why? So like, they might miss the wrong point as to why people are taking that path. So we tend to like to team up with analytics, who covers the quant side of things with our qualitative stuff. So we might see a pattern where developers say, hey, I noticed 50% of people keep taking this path, can you help me figure out why? And so we then have a hypothesis we can test against, right? We can say, well, people want to take this path. What might be the reason why they want to take that path? And so we come in from the qualitative perspective. And the reason what Dawid was saying, it's much easier to do that after launch because to do quantitative stuff, you need thousands of players to do it in any type of good way, right? Like you can't do it with just like six people, you got to do it with a lot of people. The work we do tends to be able to be done with smaller groups of people, which is why we are great to use during development because: very difficult to do a playtest with 2000 people when you're trying to get to alpha, right? Like that might be a little difficult to do. A lot easier to say, hey, can you guys get six people and help me figure out this problem? So we can. And so that's the reason we kind of focus on the qualitative aspect and not that quantitative.

Paweł B: How do you guys ensure that you have a diverse player base in terms of players that play games differently? Because we know there are more casual gamers, there are gamers that are really hardcore. There are people, for example, if you're researching a stealth option, I'm the opposite. I hate stealth, so I always go guns blazing and I try to break it until I keep hitting a wall with my head. And then I'm like, okay, I need to be more stealthy about it. But how do you ensure you actually have, different data points, different walks of life and stuff like that?

Paweł M: Especially having this small sample from 6 to 10 people, right?

Dawid: I mean, I would say that it all goes back to study design, that every study has a goal. So every single time we're designing a study, every single time we're talking with designers, we are asking them, what do they want to achieve? What do they want to know? So if this is the case that we would like to know more about stealth, or we are doing some balancing and we would like to know whether the game is fun and enjoyable for people on the hard difficulty mode with many different play styles, this is information that we take when we go to our database of players, which we take from the playtesting program, our playtesting initiative. And there we have our players profiled inside out. We know what games they play, we know what difficulty levels they like, we know their age. If it matters, if, for example, we would like to run a study on Gen Z, Gen X, millennials, we can do it. We know a lot of things and then we invite specific players to specific studies. If I want to test hard, I will not just go to the database and randomly select players, because then the insights I would give wouldn't be really reliable. I need to handpick players and select those who match the criteria that we have initially established. And then as I said, because we have those objectives every single time, and we run a lot of studies starting with individual features and then larger components, entire game, we go through all those aspects very often. We do approach them from many different angles. So over the course of development, many of the perspectives, many of the backgrounds, many of the experiences are naturally covered. And that's sort of dictated by the development of the game and what we're currently working on.

Paweł M: It does happen that you test the same mechanic let's say the stealth, right, a few times in a row. So you do the research, something is being modified and the research again. Do you invite the same players?

Josh: No.

Dawid: Well, for cases like that, most of the times, I would say 99% of the time it's always better to go for new players, for fresh eyes. And that's exactly the same reason why developers are not very good playtesters, because the fact that you already experienced that, even if it was working differently in the past, it already makes you approach it with some assumptions how it works, how you should use it, and this biases results.

Paweł B: Now I know why you haven't invited me to come back to playtests. I knew it, there's something for it. There's something in it. Yeah, you already did it. Next, we need someone different. This guy? No, no.

Dawid: I think what's really important is what Paweł mentioned that if we are testing the same thing—so if we are testing something else and the previous experience will not affect the playthrough, the playtest. Then nothing really stands against inviting the same people.

Josh: It's a very particular reason, right? Yeah. There might be a weird reason.

Paweł B: I always apply. I'm like, maybe they'll pick me this time. Maybe, maybe. But I'll be trying.

Dawid: But we've had many participants in the past who visited us, several times on many different occasions. You also need to be a little bit lucky with the amount of people we want who would like to participate in the playtest.

Josh: Tends to fill up within like an hour, like people—

Paweł B: Super quick.

Dawid: All, you know, constellations have to be in the right position for it to happen. But yeah, essentially that's what happens. And that's why we go for fresh players every single time, because this is the only way we can actually properly evaluate the system.

Josh: It also runs the risk if you bring in the same people. And once again, there are times that you do want to bring in some people, but you run the risk of them actually giving you a referendum on the changes rather than the game itself. Right? So if they saw the mechanic previously and they saw that their input changed it. That biases them, right? So they tend to look at it differently. It's much better to get the fresh eyes in. However, that does mean that quite often we have to make sure that people are familiar with it so that we don't accidentally test it as a tutorial every time someone comes in. Because if they're not familiar with it, if we ask them, do you understand how to do this within the first 15 minutes, they're probably going to say no. And so if we do that, then we're going to end up with a overly obvious tutorial that's going to hold hands to people. And it's not going to feel very good because we're not mitigating for that.

Dawid: What you're saying, I experienced that firsthand in the past. Like, I was testing a game with two separate groups, the same build. The only difference was that one group was fresh. The other group was people we brought back from the previous cohort and the group who was invited. Again, they gave a much more positive feedback, not because they truly enjoyed it more, but because they were happy that, there was some change implemented based on the feedback they gave. So they were happy that they were listened to and they didn't really focus on the game itself.

Paweł M: Biased.

Dawid: Exactly, exactly. That's why the second group, the newcomers, gave much more valuable feedback and much more impactful suggestions.

Paweł M: And does it often happen that you are getting completely opposite feedback? Conflicting. And how do you handle that?

Josh: That definitely happens. I mean, it goes back to, once again, we're not trying to say that this one person's feedback needs to be acted upon basically because of their feedback. We want to understand why they said it. So a lot of times you can actually find it really interesting that you'll get conflicting feedback. And when you kind of drill into the whys, you understand that it's actually common. There's a commonality as to why they had those differing feedback. That's not always true, but, so if you look at more the reasons why we try to rule out everything, every factor, every variable that could cause the person to have a conflicting feeling like conflicting opinion. We want to understand this person perfectly. We're not going to understand them perfectly, but we're going to try our best to understand them perfectly. And at the end of the day, Dawid and I put our hands up and we go, I can't figure out why this is conflicting. We'll most likely just not bring it up. Like we'll bring it up and say, this is inconclusive. We cannot tell you. If you want, if this is something you're passionate about, knowing we can do another study that is focused specifically on this so we can address it, but we can't figure out why this happened. And so we'll just say don't do anything based on this. Or strongly recommend, you can do whatever you want, but strongly recommend that you do not act on this conflicting feedback because we can't tell you why it happened. Which does happen. Once again, people are weird. People are funky, funky systems. It can be hard to kind of predict what's going to happen.

Paweł M: You need to be like a psychologist sometimes.

Josh: Yeah. I mean, it's a big part of the job is understanding why people think the way they do. Cognitive psychology and behavioral psychology are really important to the work we do.

Dawid: Actually, I'm a psychology graduate, so one of the very good backgrounds for, to work.

Paweł B: I think you need it. For sure.

Dawid: It is a common background, same as, social sciences in general game design. Also, like anything that really helps you either understand the player or understand design and then connect the two and draw conclusions would be a good choice if you want to become a user researcher.

Josh: One thing, like when researchers ask me like, what does it take to be a user researcher in games and stuff, I tend to tell them that there's kind of three spheres that they need to work on to be able to work in the industry. The first one is Dawid talking about kind of like being able to be a researcher. So that's the psychology of it. But that's also the scientific understanding what makes good research, right? So it's not just throwing people in a room and saying what happened—

Paweł B: Collecting data.

Josh: Yeah. You have to be able to understand what are the ramifications of how you're setting it up. Right? And so understanding research. The other thing is understanding game design and development because once again, we're applied research. So it's not just about doing good research. It's about doing good research that developers can then use. The final one is the one that I think gets forgotten the most, but it's the boring one. It's the administration one. So it's like learning how to work within a company or a corporation. Like, you know, you have timelines, you have to be able to predict how long it's going to take something to do something. There's project management. Those are the ones that we don't really want to think about or talk about, but they make or break your ability to be a good games user researcher because the game's not going to stop because you need an extra week to do research. Like you have to be able to work within that. And so researchers need to be able to kind of shine in all three of those. And that's what Dawid and I look at when we're interviewing new researchers or looking to expand the team. We don't want to pick someone that's really good in one, but can't really do the other because that's not...

Dawid: They are going to have a hard time.

Josh: They're going to have a hard time, right? Like it's going to be very hard to be able to keep up with the pace of how game development works. You need to be able to do those things. So yeah, that's.

Paweł B: I was going to ask that question, what do you need to do in order to become a UX researcher? But you kind of answered that perfectly for me. You also talked about implementation.

Paweł B: I'm going to probably throw two more questions and we'll probably wrap it up. How long does it take from the findings to kind of get into an updated feature, changed feature? Or like when, when we talk about implementing the stuff that you find.

Dawid: This time it's my turn to say it depends, but it's really that. So like, depending on what we are testing and how big the feature is, it can be one week. One month. You know, we've had cycles where we were testing the same things every couple of months because it was a big and complex topic. Very often it would be something involving story quests, pacing of the game. Those are not really easy things to change and to fix. And they very often require a lot of departments to chip in and to work on the topic to actually address them and to change them. If we are testing something small, if we are testing something, if we focus on one specific thing, then it can be every single week. We've had an iterative cycle where we're testing the same thing every single week. It was actually for Cyberpunk on Nintendo Switch to where we were having playtests Monday and Tuesday, then Wednesday was for me to analyze the results and to sync with designers and then Thursday, Friday to implement changes for the next cycle.

Paweł B: Super quick.

Dawid: Exactly, exactly. But, you know, it depends on the project. It depends on the needs. In that case that was possible. So it was also great because it allowed us to experiment much more. And if you have the cycle of, iterative cycle of several months, if you're changing the pacing, then you need to be very careful about the decisions you're making. Because if you make the decision now, then it is likely that you will not be able to go back later on. So you need to be more sure and more confident about the results and about, the data points you're using. If you can iterate frequently, then you can give yourself more room for experimentation because if it won't work, we can scratch it and start all over again. You know, go in a completely different direction next week.

Josh: This is why it's really important for at the very beginning when we ask, like, what do you want to test but also what are you going to do with it? Like it's not just about testing for knowledge's sake. We need to understand that. Are we going to give you feedback that—if it's something that you need to make the decision within a week, we're not going to give you the same feedback that we're going to give you if you have two years to do it right. Like we're going to give you what you can work with rather than put salt in the wound or something like that. We're not going to do it in a way that doesn't help you. And so the way we give feedback or the way we expect, kind of like implementation, changes on what phase of development we're in, if we're almost to the end, where this is a huge, hulking, titanic ship, where even a small turn is going to take a few months to be able to do it—we understand that you're not going to get it done next week. Like we get that. But you know, if it's like Dawid said earlier with paper prototyping, where literally the change is taking a pencil and changing it on a piece of paper, we see that happen very quickly. So earlier, quicker iteration cycles. Later, we get it.

Dawid: It's a benefit for the designer if we start earlier, because then they get more time to experiment and try out different things to fail. And that's how you learn through failure, through repetition. Trying again, learning from past mistakes. There's no reason to wait until the very end.

Josh: Yeah. It's never fun when a designer comes to ask. And then they have a realization that actually I can't do anything. Like, I want to know this and I want to make it better. But even if I find it out, there's no way they're going to let me gut this and do this over. Like we can't do it. So at that point, it's more like, okay, we can help you cushion it, right? We can figure ways to make it feel better, not change it or gut it and do it completely over.

Paweł B: Yeah. What's next for user research and user research experience? Like what the future holds, tell us.

Dawid: Yeah. So recently, a few months ago, we announced that we expanded the testing program to the US, to our Boston office. Uh, so we obviously operate there, but still Warsaw remains our main hub, main testing facility. So in the future. Not sure how close it's going to be, but we definitely would like to expand. We would like to have a facility in Boston, which is similar to the one we have here. Also I feel that, we're slowly, we are suffering from our success. Like, we finally managed to convince the unconvinced for them to come to us and request studies and, people being eager to use user research to the point that we need to, decline some of the requests because we don't have the bandwidth to deal with everything. So improving the processes, making sure that we are not wasting time on things that we shouldn't be wasting. I don't know, expanding the research operations field, getting the research operations specialist, somebody who will help us streamline things so researchers can focus on what they are hired for: research. I think that will definitely be a big theme for both 2026 and 2027.

Josh: I think kind of going along with that, with both eventually opening up a lab in Boston and then also with research operations, do more research. Like be able to not turn down stuff when it comes up, right? Right now, we're trying to increase our cadence of how many studies we do per month and things like that. And, you know, we're doing good right now, but we want to be able to do more. And when we open a lab in Boston, it's not really about only servicing the games. Working in Boston, it allows us to have two places to do research so we can have—

Paweł B: Two hubs operating simultaneously.

Josh: Operating simultaneously so that we have more—

Paweł B: And different time zones. So you're pretty much around the clock almost.

Josh: Yeah. It's a really fun opportunity. It's an exciting opportunity for us. And so we're very excited for that.

Paweł B: Especially that we're doing multiple projects now.

Josh: Yes. And it's been a blast for us because of that, juggling all of them. We're currently doing, this week we're doing three studies. Which is, which is great. We're a team of four researchers and then me and so like five overall, we're doing three studies with five people so it's a lot. And it's good. We like the work and we want to do more. And so that's where we're going with it.

Paweł B: Awesome.

Paweł M: So before we wrap up the final question, if you could playtest any game in history which one would that be and why?

Dawid: That is a tough one.

Josh: I think for me it's going to be... Zork. The text based adventure game, because it's such a weird thing for us to do nowadays, right? And so the design intent for those text based adventure games and stuff was very clear cut. You expected players to have a golden path that they would take. And there wasn't much wiggle room to be able to succeed towards the game. Like they needed to figure out the correct combination of items to get to be able to get to where they need to go. And that level of design intent is really interesting for a test because there's only one right way to do it, but you're extremely limited on the information you can give players because you don't have that permanent visual information, right? And so our words are memorable. Is it easy to keep track of the inventory? Is it easy to do that? I think that would be a very fun study to do.

Paweł M: Great. Surprising.

Josh: Yeah.

Dawid: I think I would say either original Demon's Souls or Dark Souls, even though I'm not a soulslike player. I think that would be a very interesting challenge because like, it was something new. It was you know, people waiting in queues and some saying this is the worst experience that they have ever had. And then the other going back to the queue to play it again and again, I would be really curious to, at first like where do you draw the line between challenge that motivates you to push forward, to try again and again until you reach the mastery, and at which point it becomes plain frustration and you need to, change something. How do you even think about changing things when you know the game is supposed to be challenging for the player? So we're trying to fix things and trying to address the problems becomes counterproductive for the artistic vision that you have.

Paweł B: Yeah, yeah.

Dawid: I think that would be very interesting.

Paweł B: That's a tough balancing act for sure. But yeah, that's a lot of interesting data you could pull from that for sure. Yeah. Awesome. I wish someone, uh, did like a research thing for the Oregon Trail. I remember I used to play Oregon Trail as a kid. Everybody always died in my party.

Josh: Always getting dysentery.

Paweł B: Exactly, exactly. So yeah, I never found out like, what's the best way to beat that game? So yeah.

Dawid: That'd be interesting.

Paweł B: Maybe you guys can help me with that.

Dawid: Let's talk after, that sounds pretty fun.

Paweł B: Yeah, we can do it on the weekend or something. Perfect. Awesome, guys. It was great talking to you. Like I said, for me, this is one of the coolest initiatives that we started. And since I'm kind of responsible for doing studio tours, I always show where you guys are sitting and you're doing your research. And I'm like—

Josh: We love that.

Paweł B: I'm like, this is so cool. This is such a cool new initiative that we're doing. And I'm happy that you guys are doing it. Awesome work.

Paweł M: Can't wait for future collaborations with you guys. All right, great. Thank you very much.

Paweł B: Thank you.

Paweł M: Thank you for watching this episode. I hope you enjoyed the conversation as much as we did.

Paweł B: Yes, it was a blast. And as always, don't forget to comment, like, subscribe. Let us know what you're thinking about the episode. And if you want to see someone on these seats, also let us know and we'll see you in the next one. Bye!