Ep. 40 | Customer-Centric Research with EA
This week in the Accelerator: What is customer-centric research and how is it different from traditional methods? In this episode host Allison Hartsoe interviews Jodie Antypas, VP of Research from Electronic Arts. Jodie shares examples of how the customer-centric research work her innovative team delivers drives real world performance for Electronic Arts. She also talks about methods to use and how to integrate with product teams to maintain the essence of the brand while navigating customer-centric behavior and motivations.
Please help me spread the word about building your business’ customer equity through effective customer analytics. Rate and review my podcast on Apple Podcast, Stitcher, Google Play, Alexa’s TuneIn, iHeartRadio or Spotify. And do tell me what you think by writing Allison at info@ambitiondata.com or ambitiondata.com. Thanks for listening! Tell a friend!
Read Full Transcript
Allison Hartsoe – 00:01 – This is the Customer Equity Accelerator. If you are a marketing executive who wants to deliver bottom line impact by identifying and connecting with revenue generating customers, then this is the show for you. I’m your host, Allison Hartsoe, CEO of Ambition Data.
Each week I bring you the leaders behind the customer-centric revolution who share their expert advice. Are you ready to accelerate? Then let’s go!
Welcome everyone. Today’s show is about customer-centric research strategies and to help me discuss this topic is Jodie Antypas. Jodie is the VP of Research at Electronic Arts and also an audience favorite from our first customer centric conference and someone I just love talking to. Jodie, welcome to the show.
Jodie Antypas – 00:55 – Hi Allison. Thanks for having me.
Allison Hartsoe – 00:57 – So, tell us a little bit more about your background and how you were drawn to this topic because as far as I know, you know, in in college, maybe academics study research strategy, but it wasn’t as much of a professional line at that time as it is perhaps now. How did you get to this point?
Jodie Antypas – 01:14 – Yeah, it’s pretty interesting that I stumbled into this career. I think I’ve always had a real passion for understanding people and real curiosity around people, um, and the decisions that they make and product choices or situations that could really be better for customers or consumers. Even if at the time as a kid I didn’t realize that that was customer experience or that’s what it was. I have a funny memory of my sister. She’s a couple of years older than I am, so it was probably about 10 and watching her always try to swallow pills and she would gag basically anytime that she tried to swallow a pill and she’d grown out of taking, you know, the, the liquid or the chewables. And I kept watching her doing this thinking there’s got to be a better way, you know, I’m going to solve this problem for her when I get older I’m going to, you know, create something so that she can swallow pills. Um, you know, obviously that didn’t happen. Somebody else got around to it sooner than I did, and my parents didn’t want me tampering with medicine and the medicine cabinet. But um, you know, something that I’ve always kind of keyed in on is how that experience was bad for her, but how many other people out there were like her and had that same problem.
Allison Hartsoe – 02:22 – Isn’t it amazing how when we’re kids, like the things that really drive us as an adult in our career, you know, kind of surface as kids, you, you have this tendency, this natural inclination. Yeah.
Jodie Antypas – 02:33 – I don’t think I knew that that’s what that was at the time, but it ended up being kind of core to who I am and what I do today. I have another funny story. Um, my parents like to tell this story how when I was about 12, I think I saved my money and bought an alarm clock, which is kind of a strange thing for a 12-year-old to do it and save her money for her. But I wasn’t much of a morning person.
Allison Hartsoe – 02:57 – Yes. And I can vouch this because I have a 12-year-old boy,
Jodie Antypas – 03:01 – but you’re probably trying to drag out of bed every morning.
Allison Hartsoe – 03:03 – Exactly.
Jodie Antypas – 03:06 – So I bought this alarm clock. It was stony alarm clock. And a few months after I bought it, it stopped working, um, so try to take it back to the store and they wouldn’t take it back. And then I called the customer service and you know, there was nothing that they could do and somehow, I tracked down. This was long before the Internet age, somehow, I tracked down the name and address of the president of Sony America, um, and I, I wrote him a letter and I my alarm clock back in the box and I said this doesn’t work. And I had an expectation that, you know, I saved my money and I need an alarm clock and, you know, a few months later or maybe a few weeks later and new alarm clock and a letter from the president of Tony showed up at my house. But I think,
Allison Hartsoe – 03:50 – Is that right?
Jodie Antypas – 03:51 – yeah, I’ve always really been interested in, um, you know, what today is known as user experience or customer experience. Um, even though I didn’t really know what that was back in the eighties,
Allison Hartsoe – 04:02 – oh my gosh, I can’t believe you actually did that. That’s fantastic. It’s one of those things that everybody’s like, oh, well you could write a letter to the president and then nobody ever does it, but you did. That’s fantastic. I had a bad user experience and you’re responsible.
Jodie Antypas – 04:18 – That’s right.
Allison Hartsoe – 04:20 – That’s fantastic. Okay. So as a kid you clearly had an interest in user experience. How did you get from that point to electronic arts?
Jodie Antypas – 04:30 – Um, yeah, so fast forward about 15, 15, 20 years and I was working at a small consulting firm that focused on the sports industry. And so, in that role I did a ton of primary and secondary research essentially with no training and my boss would always comment on how throw my research was and how good I was at uncovering things that they hadn’t expected or things that they hadn’t known we’re going to be issues or potential issues. And about the same time while I was. I was doing that while I was in business school and about the same time I had one of my specific professors with Damian said, you should really take a data mining course. You’re pretty good at this. Then I kind of scoffed at him and like data mining. No, I’m here to do marketing. I’m thank you very much. So, um, but little did I know again that that would, um, you know, play a part in my future and after I finished getting my mba I started working at a very small research agency and we focused on public health issues, things like smoking cessation and traffic safety and healthy pregnancies and women.
Jodie Antypas – 05:34 – Um, which was pretty interesting topic to work on, but I decided after a few years of that, that I wanted to work in house on a bigger consumer brand and I joined Nintendo around the time of the we were launching back in 2007 and that was really the entry, my entry into the gaming industry.
Allison Hartsoe – 05:53 – Perfect timing there. Yup. Wow. Wow.
Jodie Antypas – 05:57 – And then, um, yeah, joined EA about three years after I’d been at Nintendo and have been at EA for the past eight years in a variety of insights role.
Allison Hartsoe – 06:07 – Wow, that’s a, that’s a fantastic path. So, tell us a little bit more about, would he do on an everyday basis or what your team does?
Jodie Antypas – 06:15 – Yeah, my team, I lead a research for EA and my team includes our consumer insights team and our user experience research team. And so, we do all of the product research, product development concept. I’m thinking about games that are coming in development, what features we’re going to put in them, what games we should have in our portfolio. We look at user segmentation or a player segmentation, um, our, our user experience research teams, we have eight labs around the world and we bring players into our labs on a weekly basis and they’re giving that feedback directly to our production and creative teams that are building the game. Um, so that’s very customer focused and then yeah, so we have, um, you know, eight labs around the world. You come in, we have producers and developers, designers in the back room watching them feedback on their games typically every week. And um, then on the marketing side, we do basically all the research that you would need to do to take a game to market. So that’s positioning or communication strategy, target audience identification, really anything that can help with a go to market campaign.
Allison Hartsoe – 07:27 – It really got the full span of customer and about customer centered research. It’s not just marketing, it’s product development. It doesn’t get it and it gets into satisfaction as well too, right? Yeah, so it’s really the full scope.
Jodie Antypas – 07:41 – Yes. We do product satisfaction research as well. We have a pretty robust NPS program that we run internally here at EA and I think one of the unique things about our team in research here at EA and the way we’re structured is it’s a little bit different than how some other companies tend to keep user experience and consumer insights separate. In many companies, they have consumer insights, focuses primarily on the marketing or the brand or communications and UX focuses on product research and we actually have our teams work more closely together and I think that really served us well because you want what you communicate about the product to actually be true of the product and true the user so you’re not just developing for one particular audience and then trying to sell it, doing completely separate research. I think those it, it really serves us well. Having the two teams working closely together,
Allison Hartsoe – 08:32 – That is a very good insight and that’s something that we don’t see day, and that’s also what happens with companies like EA that are leading the way is the org structure start to change because you end up with such a tight focus on the customer that it’s not, it doesn’t make sense to have divisions where everyone should be working more tightly together. So that’s, that’s a really great point. Tell us a little bit more about customer centric research and you know, how that research landscape may have changed from what we might traditionally think of as research. You were just talking about the CX, uh, focusing on brand and the UX focusing on product traditionally, that’s obviously one change. Are there other changes for what real customer centric research is?
Jodie Antypas – 09:18 – There’s a couple of things. In the past 10 years or so, there have been a plethora of online tools that make research much faster and make the turnaround time much faster, which is particularly important in the tech industry where we see product changes happening really quickly. So that’s really helped us. When I started my career, we were doing, you know it’s going to date me, but we were doing mail surveys and online random digit dialing surveys for public opinion polling and so things are quite a bit different now and we’re lucky here working in the tech industry that we can basically source than online sample, how to be representative of our player base. I’m also mobile phone research is really quite a bit more popular. Everybody’s basically carrying around not just a camera and a cell phone, but also a survey response tool in their pocket, so it’s a lot easier to do kind of real time quick ethnographic studies on what’s happening and where you are and what does your environment look like.
Allison Hartsoe – 10:21 – Do the online tools also prepackage the way that you look at the data. In other words, is it easy for them to capture certain pieces of information that hard for them to capture other pieces of information that might predisposition the research to tilt one way or another? So, for example, if I’m running social media, it’s easy to see a like, but alike may or may not be indicative of something I care about,
Jodie Antypas – 10:47 – Right. You must be responsible for your own sampling typically in research. So, all statistics, you must have a good clear sample. And so, sibling strategy hasn’t changed too much, or sampling theory hasn’t changed too much. Just the tools that you can use to reach the audiences have changed. So, once you collect the data, I think a lot of the tools do make it easier for you to slice and dice the data or populate charts and make it a little bit faster and make your research process more nimble. To answer your question about the gambling, I think, or one of the things you were getting at with asking about social media is we do see different types of audiences are gravitating towards different online communities. So, we have a pretty. In gaming we have a pretty heavy audience that focuses and listens to or reads Reddit. And so that’s one area where we pay attention to you, but we also know that it’s not the entire audience.
Jodie Antypas – 11:44 – So when we’re doing our research, we’re looking at those people who are on our board and on our blogs and on Reddit and on our social channels and definitely taking account their voice. But also, we have to weigh that against the entire population overlooking at FIFA. We’ve got 20 million players, we know some of them are going to be communicating with us on those online tools and message boards and, and Reddit. But we also know that probably, you know, upwards of 15 million of them are not. And we have to take into account their voices as well.
Allison Hartsoe – 12:15 – Oh, that’s fascinating. Because I think we tend to think of maybe more homogenous, you know, it’s people who are online versus people who are offline and ideally, it’s really not like that. It’s a reflection of the offline world and understanding the rich pockets where you get different voices in different areas. Just shows how much you’ve really listened to those people and what they’re saying in those areas so that you can separate it. I think that’s why people don’t typically separated. Would you agree? Is that, is that people tend to look at it in binary colors.
Jodie Antypas – 12:53 – I think many other industries, they might, we absolutely cannot because you said the online, the consumer online and so there’s different groups within that online consumer base who have differing opinions and different engagement with a product we can, we can typically slice our consumer audiences by engagement in terms of how many days they’ll play our game and we see vastly different every anywhere in FIFA from 300 plus days of playing the game per year to people who buy the game and play it for, for just one day. Um, so with that, we know we have, we have to basically cut our audience data by engagement as one view of what we’re looking at.
Allison Hartsoe – 13:39 – But that also implies that you’ve connected all the data. Was that a big challenge to performing research to be able to do your job effectively?
Jodie Antypas – 13:47 – Yeah, that’s actually one of the newer innovations in researcher or newer skills that we have in research is tying the attitudinal data, so typically survey research is about what do you think. We used to ask a lot more about what people did in surveys. We used to ask a lot more about behavioral data, you know, what product did you own and how many days did you play it and who did you play it with? Now we don’t have to ask those things because we can see that in our telemetry or online data of what people are actually doing in our games and so it really helps us to be able to marry the behavioral data, kind of the what’s happening with the why is it happening.
Allison Hartsoe – 14:26 – That makes perfect sense. Tell us a little bit about the kind of person who plays games with Electronic Arts. Are they typically coming back once in a while or are they really embedded? How do you think about them?
Jodie Antypas – 14:40 – I don’t think I can answer about the typical person. That’s a little bit like asking about the typical American.
Allison Hartsoe – 14:45 – Good for you.
Jodie Antypas – 14:46 – We have so many people, we think we have 300 million players that we know we engage with and there’s no one way to describe them. So, we have people that are really, you know, engaged in one particular game and we have people who are coming in and playing our subscription service and replaying throughout many games. And so, one of the things that we’ve focused on in the past few years is looking as we have access to better data and have access to the telemetry and the behavioral data as we’ve been able to look more closely at the different audience cohorts in different segments. And we’ve been spending a lot of time looking at target audience identification and really focusing on who are the players that come in early, who are the players that come a little bit later and what are, do these different cohorts in different audiences want so that they will stay engaged in the game. Probably about 10 years ago, EA has focus was really on selling games, um, and focused on selling games to the consumer.
Jodie Antypas – 15:42 – And one of the things that we’ve learned over the past few years is that we really want players to be engaged in our games. We want them to come back and experience our games and play it and have a good time. We know that when they do that, they’re more likely to purchase the next game and they’re more likely to get their friends to play our games, um, and it’s better for us and it’s better for them. So, when no longer are we just trying to sell games to consumers and have them buy a one-time purchase, but gaming is much more about a relationship. Um, and then an ongoing experience that we want them to have.
Allison Hartsoe – 16:12 – I absolutely love that comment. I cannot tell you how many conversations I have with marketing teams who are still trying to respond to you just sell the product, sell the product, and they still think about it. And it’s like fingers on a chalkboard for me when somebody says funnel, because yes, there are funnels online and we do analyze funnels, but the concept that I’m just going to shove a whole bunch of people through a narrow funnel and if I get enough reach then I’ll get enough conversion is so outdated and what you just said is really about pulling about playing the long game and pulling people through from the back half of the satisfaction. That continued engagement. I love those measures. I’m sure you have some examples around, um, how, how you’ve done that or how you’ve executed that with different products at EA.
Jodie Antypas – 17:00 – Yeah. I think one of the examples that I can talk about products and about five years ago we started to see a lot more growth in North America as a result of some of the marketing that we were doing. And so, you know, that the objective was go out and understand these new players that are coming in. And that led to some really interesting insights about them and how we can change the game and change some of our communications to drive their engagement a little bit higher. So, the first thing we did when we were trying to learn about this audiences, we started with ethnographic research, so we went into their homes and we asked in some cases that they invite their friends over the people that they would usually play with. And we just watched them. We observed them. And then we talked to them after their gaming session. One of the things that we found was that in these groups of newer players was typically one kind of football focused fan. Sorry, when I say football, I mean soccer, um, one, one soccer focused fan.
Jodie Antypas – 17:59 – Somebody who had some connection, whether it was a family member, or you know, family history or particular fandom of a team typically in Latin America or Europe. And this one friend was really rallying the rest of the group to get engaged or introducing them to FIFA. And because the game is pretty easy to pick up and play, it allows larger social experiences up to four on four. Um, it’s actually pretty fun for people to watch if you’re not playing. And there’s low barriers to entry in terms of understanding what’s happening on the screen. But we found that the newer players didn’t really know a lot about the sport and oftentimes they thought they did. So, I remember vividly being, I’m in an interview with one participant who said, oh, I’m a huge soccer fan. I watch every week. I play the game. And then a few sentences later said that he didn’t really understand all sides.
Jodie Antypas – 18:52 – And that was a key insight for us that even if these players were telling us that they really understood the sport and they were huge bands, the ones we dug a little deeper, we could tell that they didn’t. So, if you ask any, you know, hardcore football fan in England, you know, if somebody doesn’t know offsides, um, you know, that’s not even a football fan, right? They shouldn’t even be allowed in the stadium. So, what we learned was that one of the things we needed to not just onboard them to the game, which is pretty common in video gaming, teaching people which buttons to push and, and what to do as they’re learning a new game, but we needed to better onboard them to the sport. And this finding allowed us to dig into our, our quantitative research and really slice our data based on knowledge of soccer. And then we could see different behaviors like, you know, if you had low knowledge of soccer, you are less likely to play in multiple modes in the game. You were less likely to play, um, in our FIFA ultimate team mode, which is a pretty sticky experience which keeps you coming back, you know, typically multiple times per week.
Jodie Antypas – 19:56 – And so what we wanted to do is teach them through the game more about soccer, given them a more of a connection to the sport, to a team that they could become a fan of or a player that they could become a fan of so that we could get them more engaged in the game. And so that they would have a better experience.
Allison Hartsoe – 20:13 – I love that. It sounds like taking all of the ethnographic research in and all the commentary and you probably had a number of different nuggets that came out of that that formed a hypothesis that would have said something like if the knowledge is low, then we suspect that the engagement mail to be low. And so you decided to slice the data based on that insight. Is that what typically happens in the way you conduct research is, is that the methodology where you of picked different. I call it the tip of the spear, you know the way that you’re going to slice data in order to see the insights open up?
Jodie Antypas – 20:55 – Yeah. We typically start with hypotheses even in the qualitative, the qualitative sessions of our work. So, we kind of know where we’re going out there and looking for. In this particular project, one of the hypotheses was around is there a lot more play on college campuses and mom that 18 to 24 population, so we did some work with those, but we require that our researchers, when they’re taking on a new project that they work with their business partners, their product development partners or marketing partners on what are the hypotheses, what are the things that we’re trying to understand because that really leads us to much better research. We can be much more focused on what we’re trying to achieve and then when we come out of qualitative research we do. We form those hypotheses. We look at all the insights and then we will make sure we have the right questions in any form of quantitative research or that we can cut any existing behavioral data that way so that we can then kind of dig into those and then it becomes a little bit like a flower. So, you dig into one piece and then it opens up a little bit and then you see something else that you might want to dig into.
Jodie Antypas – 21:58 – So it’s maybe a combination of art and science where you’re looking for those nuggets and those insights and then you have to kind of keep drilling down a little bit deeper until you find something that’s really interesting that you can then act on.
Allison Hartsoe – 22:09 – I love that example. It is exactly like a flower where you keep opening it up and trying different angles. Now, and I don’t know if it pertains so much to this example is to other examples, but the obvious question for me is how does that help you avoid confirmation bias? Which is something we just start with all the time in, in data analytics.
Jodie Antypas – 22:32 – Confirmation bias is a good one. I think having objective researchers, um, is really important and sometimes one way around this is having researchers as opposed to your product development leads through your creative directors or your market is leading the research. They certainly can if you don’t have the staff to do that. But, um, you know, we find that the researchers are typically a little bit less, you know, married to a particular idea and they’re a little bit more willing to be open minded. And that’s our job and that’s our nature is to do that. So, we always encourage people to just listen for what you’re looking for, but you know, listen to all the consumers or don’t just go out and talk to the one person who’s like, you talk to a whole different set of players. And that’s one of the things that we ensure that we are talking to a robust or a diverse audience in our research. So, we’re not just talking to the people who look most like the, uh, you know, the creative director or the marketers were talking to a broad set of players.
Allison Hartsoe – 23:28 – And it sounds like you’re talking to a broad set, defined in different ways to find based on the knowledge of soccer to find based on how many products they’re using or how many days they’re engaged or it seems like there’s a lot of different ways you can slice that group well beyond what we would typically think as diversity, which is demographics slicing.
Jodie Antypas – 23:48 – We look a lot at behavioral diversity or attitudinal diversity. So, you know, are you interested in a soccer game? Are you interested in this shooter? Um, are you interested in a simulation game? We looked a lot at um, or motivations to play. Why do you play games? I play to express myself. I play to compete with others. We know that there was diversity and motivations of why people play games as well.
Allison Hartsoe – 24:11 – I love it. So, do you have another example of different ways that you’ve used the research to come to an insight?
Jodie Antypas – 24:17 – Yeah. Another example is our battlefield one game. So, this was a world war one game that launched a couple of years ago and what was really interesting about this research was we focused a lot on how we were going to bring the campaign to life and get people really excited about playing the game. And when we first went out to learn about how we would do this in our communications, our marketing strategy, we found that players didn’t really know a lot about World War I. They weren’t particularly educated on it, not just in North America but kind of around the world there, the lack of knowledge and they weren’t really interested based on the perceptions that they have. World War I. So, they thought World War I meant, you know, flow gameplay, trench warfare, in some cases they end up, even though we know that’s not true, I’m not enough progression, limited weapon variety or not enough vehicle. Basically, anything that our fans tell us makes the battlefield franchise exciting and interesting. They didn’t expect to have existed in World War I.
Jodie Antypas – 25:18 – So it was a big hurdle. We knew from the beginning that we had to control the message of the campaign and that can actually be really difficult in the video game world where people that are so excited for the products that are coming out, anything that leaks even just a tiny bit has the potential to go viral. So, we knew that it was really important for us to get out ahead with the message and the right message and we had to be really clear on what were the benefits of this game and why should you be excited to play it. Even if you think that World War I, it’s going to be really boring. And one of the ways that we did that is we started talking about the game slightly differently. So instead of talking directly about World War I or what was happening or what the battles where we started talking about the dawn of all-out war, or no battle is ever the same or war on an epic scale. Those were signaling to players, but this is something new that they hadn’t played before. Epic scale with a huge battle, which is pretty appealing to players in a shooter game.
Jodie Antypas – 26:16 – We weren’t hiding that it was a World War I game, but what we were doing with teaching or showing them and telling them all of the ways that this game delivered for them, um, what they wanted out of the shooter. And it happened to be set in world war one. World War I was the backdrop for all of these great experiences that they were going to have. We also learned that our video assets and our trailers, we’re going to become really important. So, we had to show people not tell them what this game was going to be about. So, if you look at of our first trailers in our earliest communication about the game, you’ll feel a big variety of things happening. You’ll see things like horses or trains or that went and things that hadn’t previously been released in other sugar games, so it was, it was new experience and we’re really demonstrating all of these things. It could be new and exciting to players that happened to be also a world war one game.
Allison Hartsoe – 27:05 – So how did you come up with these ideas though? I mean it sounds like you, you, you use the research to come to these ideas and then you found out that they were effective. Was there some agile process along the way that kind of led you down this path? As we were talking about before, like the flower unfolding?
Jodie Antypas – 27:22 – With this one, we worked pretty closely with our marketing team, our, our ad agency partners and our internal advertising team. And you know, we started with uncovering the insights with like, oh no, we’ve got a problem. That was our first round of research with, oh no, we’ve got a problem when we talk about World War I, there was no knowledge there and it’s going to be a barrier if we don’t communicate that in the right way and getting the team onboard with that as the kind of key strategy was pretty important. Then we actually had a couple of different rounds of research and as I mentioned, we were a little bit old school and doing this because of the leak potential, so we didn’t want it to get out online, so we actually did all of this research in person, which I don’t always advocate for, but in the gaming industry when you’re trying to keep something secret, sometimes you have to go back to older methods. So, we did a ton of impersonal research and bringing people into focus groups or one on one interviews and would show them the gameplay and ask them questions about it. We would give them different statements to react to you in which one sounded most exciting, which one sounded like it was a fit for World War I. Which one didn�t, and we were really doing just multiple rounds of qualitative research until we felt that we really honed the message and it was something that we could go to market with.
Allison Hartsoe – 28:36 – That makes sense. Why not do in-person research, what is the negative side of that and that more traditional approach
Jodie Antypas – 28:44 – In person research is great for certain objectives. It’s typically really expensive so I can interview somebody over Skype for a lot less cost than I can bring somebody in person. Me and my team flying to a research facility or online interviews or cell phone interviews are just much more cost effective and you can turn it around much more quickly. You can turn the data around much more quickly, but in our case, we didn’t want to take any risks with putting any research stimulus online. You know, as much as we want to trust our players. You never know who’s sitting on the other side of the screen with a cell phone ready to capture anything and then posted on reddit. So, we chose the slightly slower, slightly more expensive method. In this case for 100 percent security.
Allison Hartsoe – 29:31 – That makes sense. Yeah. I follow that logic. Do you think when people come into a physical research environment, it shapes what they’re saying when they walk into the room, do they do they behave differently than they might behave at home and do you have to account for that?
Jodie Antypas – 29:47 – We do. And I think a lot of good qualitative research is really dependent on a great moderator or somebody who’s leading the discussion and a great setup, so making sure you understand what you’re trying to achieve and what’s the best methodology for doing that. You know whether that’s going to be a one on one interview might be more effective and doing group interviews. You want to go in home when you want to see how somebody is using something in their real life and really get a context for how they’re using your product. If you’re talking about concept research and how somebody might react to something, I think, okay, to do that in a focus group, in a group setting, but you really need to think through all of these situations. We see often you go out and hire a qualitative research agency. They want to just, you know, throw eight people in the back of a room and do a focus group and I would really encourage people to think about what is that the right way? Is it better to do a one on one or we do a lot of friendship pairs in gaming where you bring people in with a person that they play within their real life.
Jodie Antypas – 30:44 – We’ve actually done some, what we call dueling groups where we’ll bring in two different types of players and they don’t know that they’ve been typed as different based on a series of questions that we’ve asked them and then we forced them to have a group discussion, which is really interesting that at the end of the group we tell them, okay, like you guys weren’t agreeing because you know your different types of players and that leads to some really interesting discussion.
Allison Hartsoe – 31:04 – Wow. I bet that comes out with very interesting insights about the attitudes and motivations as people work through those discussions. That says that’s a fascinating concept of friendship pairs. I don’t think I’ve ever heard of that.
Jodie Antypas – 31:16 – Yeah. You know, there’s just some really good ways that you can make people feel comfortable in their environment where they do want to tell you how they’re feeling and they get to be in a little bit more of a natural state and sometimes being with somebody that they know really well it can help bring that out for them.
Allison Hartsoe – 31:32 – Absolutely. Well, we probably have time for one more example if you’d like to give any others.
Jodie Antypas – 31:36 – Sure. So, I think one other example I can talk about is how we’ve used research in a situation where we didn’t watch a game so effectively, and we have a couple of examples but one that I’ll talk about is the sim. So, the Sim for is, you know, a game where you can create lots of different characters. I think many people are probably familiar with it and we launched The Sims back in 2014 and unfortunately it didn’t, didn’t get a lot of great critical feedback from reviewers and from the community. Um, mostly because of some of the decisions about what to put in the game. We did a complete overhaul of some of the main game systems, the game technology. It had a great new building system and a great new creative mode since we’re much more emotional, which led to great stories. But from the community standpoint, they were looking at the game saying there’s not as much content in here that within the three had been built up for many years.
Jodie Antypas – 32:33 – We’ve been releasing content for probably the last five years on this industry. And that’s what people were comparing it to. So, you know, we went out and we got a lot of critical feedback from our community and the development team in a lot of time with our research team, figuring out what know players wanted most and we were able to then patch the game multiple times over multiple years and they stayed with the community and listen to the community pretty regularly through social listening, but also through more formal research channels, um, and our NPS program. And that allowed them to continue to kind of build that credibility with the team to now, I think earlier this summer we announced that we sold 10 million copies of them. So pretty healthy community today where it didn’t necessarily start that way.
Allison Hartsoe – 33:20 – Yeah, I can see the value of staying with it over time. Did you have to sell that internally? Was there a pressure just to ditch it and move on?
Jodie Antypas – 33:29 – Fortunately, the students had been a really great franchise for EA, so people weren’t really willing to wrap the game and move on. Um, there was a really great community based that does love everything since. And so, we were able to get some people onboard pretty quickly, but it really didn’t take a lot of empathy from the development teams and the marketing teams to say, okay, we’re gonna to and we’re going to listen and we’re going to try to hear you. Um, how do we communicate that to you while also making sure that communities know that sometimes, you know, game teams have to make really difficult decisions. Like, you know, how much content can we put in? What are we choosing between how are you prioritizing our resources because we can’t just build everything.
Allison Hartsoe – 34:07 – As you were making those slow and steady changes, did you find that you were. Were you always moving forward or was it like two steps forward? One step back, always tinkering and trying to get the right mix. Let the Rubik’s cube in a way.
Jodie Antypas – 34:20 – I think there were many ways that we were trying to communicate with the community that we were listening to them and trying to add as much content as we could and listening to them. I’m sure that we didn’t get it exactly right the whole path, but I think we were able to demonstrate to the community is that we were listening, and we were putting in the content that they wanted. Particularly pools and a couple of years later coming back out with toddlers and so giving them this content for free made them realize that we weren’t trying to just earn more money for them, but it was part of our development process and something that we needed to be able to do over time to get them the robust content that they were looking for that they had in this industry.
Allison Hartsoe – 35:02 – Imagine you’re communicating that back out and helping them know that good things are coming, hang in there and they continue to engage. That’s a. that’s a really great way to honor the community and to listen and feedback what you’re hearing. So, are there particular customer centric methods that you find? Are Your, the screwdriver, the hammer, the go-to methods that really do a lot of heavy lifting for your customer centric research?
Jodie Antypas – 35:30 – I think there’s a couple of things that we do. One is our brand research, really understanding the essence of what our brand or in our case, our game being two players. So, we know that for the center for example, it’s really a lot about creativity and expression and we know you know what that means for battlefield, it’s about authenticity and it’s about great moments that you can’t experience any other game. We talked to a lot of players and all, they’ll talk about this only in battlefield moment, so really understanding what is it about our experience and about our game that makes it special so that we can infuse that into our other product research and make sure that when we’re coming up with new product plans that those elements are there and that they exist and then how do we talk about those things and it really becomes tricky when you’re trying to deliver on a brand essence or a game essence while also creating new innovative and different experiences for players which is really required and demanded in our industry, which is so fast moving and changing.
Jodie Antypas – 36:30 – And then the other piece of research that we focus a lot on is our NPS research. So, say what you owe about NPS. I think a lot of people love it and other people don’t like it so much as a metric. I think it’s been really effective at EA because we’ve gotten people to pay attention to the player and so we have both an NPS score, we have a number and then we have a really robust in house made text analytics tool and program that we really dig into. We typically get about 10,000 responses from players. Everything from, you know, the terrible reasons that everybody hates us and everything we did wrong from the game. Even if they’re playing it every day to people that are just gushing with how much they love the game and then kind of everywhere in between. And we really dig into this text. We send reports out to our players. We’ve sent, you know, text files to executive where they’re reading player comments directly. And so, I think, you know, these two pieces of research where the brand research really helping set up the project from the get-go and our nps research on the back end when we’re looking into product and customer satisfaction really work well together to keep us focused on the player.
Allison Hartsoe – 37:36 – That’s a really great insight that we oftentimes talk about trying to pull through the voice of the customer. But what you said that was so interesting is the executive level attention to that and how it pivots the company to pay attention to the player. Uh, and then coupling that with brand research, which I actually haven’t you, I’ve heard obviously about brand research before, but the way that you anchor into the brand, sometimes we see customer centricity as a third part of the equity picture. So, we see brand equity, operational equity and customer equity. But the connection here between brand research and customer and hooking that into NPS I think is fairly unique. And I haven’t heard of anyone else doing this. I love that. Focus on what’s unique and keeping that, um, before you go into the promise of what is the innovation and where should you go so that you never really lose the essence.
Jodie Antypas – 38:35 – Yeah, I think it worked well for us. We’ve had a few missteps along the way. We’ve had games where we’ve tried something new and we had the brand stray too far away from what it means to players and we’ve learned along the way. So, um, something that we’ve been focusing on more so in the last couple of years is we’ve learned how to become more player centric.
Allison Hartsoe – 38:54 – But you catch it quickly and that’s what’s good is because you’re always listening and uh, there are many brands we can think of where they stray from their brand essence and they don’t catch it quickly and all of a sudden, they realize the customers aren’t there anymore and they start asking themselves why. And by then it’s a little bit late, a couple of years down the road and, and, uh, you know, sales are down at that point. Customers are defected, and it must be very, very difficult to win back people once they’ve gone.
Jodie Antypas – 39:24 – Yeah, and I think we’re looking at different metrics. We have core player metrics that we focus on and we’ve set the metrics in place to be able to track that a little bit more closely so that we don’t get too far away, and we do trend to notice if our players aren’t coming back or if they’re not engaged on our experiences.
Allison Hartsoe – 39:43 – Can you talk about what those core player metrics are?
Jodie Antypas – 39:45 – I’m sure so we have four core player metrics. One is NPS, which really focuses on the player feedback on our product. One is the number of unique users that are playing a game that looks at kind of the scale and volume of people that are coming into our franchise. One is session days, and this is the way that we look at the number of days people are engaging with our product and how many times they’re coming back. That’s one of our core measures of engagement and then the last one is average spend per player and that’s looking at how much people are spending.
Allison Hartsoe – 40:18 – That’s a great focus for the company. I see a lot of companies that kind of stray around and one of the things I really admire about electronic arts is the rigorous focus. Two core metrics and which is probably the topic for a whole other show, but I imagine that that helps focus your research because when you know what matters, then you can slice and dice the research to get to those goals and if you have a very nebulous corporate focus or a very nebulous corporate strategy, then it can be very hard to figure out what you should be surfacing and the research. Is that right?
Jodie Antypas – 40:56 – Yeah. I think one of the nice things about corporately or metrics, but they’re focused on the player, so what you were just saying as many other companies don’t necessarily have the right metrics in place and we’ve been able to focus our metrics on the player which takes the conversation away from ame review or what had metacritic say about our game or the overall revenue of the company. Which is really important. But when we’re focusing on the player, it takes the conversations back to, well, how much did the players like the game, you know, how engaged are the players and how many players do we have in our games is really our source of truth that we can the company focused on.
Allison Hartsoe – 41:34 – Yeah, truly customer centric. Well, let’s say that I am interested in launching our research program and trying to get more customer centric research into my company. What should I do first?
Jodie Antypas – 41:48 – One of the things that we advocate I talked about earlier was asking the right questions. So, we have our researchers work with our product partners and our marketing partners to really say what are the hypotheses? We have them actually fill out a form and we say, you know, you need to fill out this form. People kind of laugh at us, but it’s really helped us deliver research that is strategic in nature and it gets everybody aligned to the objective. All too often we see this is probably pretty common in many companies that people will ask, and they’ll be like, oh, we’re going to do survey. Asked a bunch of these questions and it’s not very thoughtful from the outset about how you want to use the research and what you want to uncover, and it isn’t really aligned to the decisions that need to be made about the business. Who are researchers focus a lot on what are the business questions, you know, don’t come to me with research questions and try to create survey questions or focus group questions, but come to me with what you need me to solve for your business.
Jodie Antypas – 42:42 – I think being open and empathetic to your players or your customers. And we talked a little bit about avoiding confirmation bias, so don’t go into research looking to confirm what you think is right, but also make sure that you’re listening to the player and that you’re understanding what their truth is. So, there’s so many times I can remember being in the back room of a focus group with a creative director and having them say, you know, like we’ve got to go in the room and we have to tell. Tell them that they’re wrong. That that’s not how the product works and the game that’s in the game, we have to go tell them. And what I tell the people in that room is no, we need to listen. We’re here to listen. Um, and that’s their truth. They may be completely wrong, but that’s because we haven’t done a good job communicating it to them or they don’t know everything that’s in our game. And so, while the customer may be wrong, what they’re telling us is what we need to hear. So, go in and be open and empathetic and don’t leave the focus group thinking that the players wrong or they don’t know what they’re talking about. Um, because that’s not gonna get you anywhere.
Jodie Antypas – 43:43 – And then lastly, I would say empower your researchers and really bring them into your business conversation. So, in many companies, research is more of a support function or you know, you’re just there to write the survey questions or you’re there to deal with the research vendors and then coordinate things for us. But if you, you treat researchers like a QA function, you’re really limiting their ability to get to great insight by understanding the business problems and what you need them to solve for you. So, the more you research or knows your business challenges, the better we can design research and get to great insights with your audience.
Allison Hartsoe – 44:18 – I love that. It makes perfect sense. Here’s an area of specialty focus that you knew how to conduct research, but what you don’t know is the detail of the business or the product and that’s the partnership is what is the problem to solve and helping them work through that. That’s very good, Jodie. So, let’s, um, let’s take a minute and recap a little bit about what we talked about on the show. But before we do that, if people want to reach you, how can they get in touch with you?
Jodie Antypas – 44:47 – You can find me on Linkedin.
Allison Hartsoe – 44:49 – Oh, that’s easy enough. Do you want to go ahead and spell your last name just for people who might not be able to find you?
Jodie Antypas – 44:55 – Oh, I’ll give it a shot. It’s a little bit difficult to spell out loud, A as in apple, N as in Nancy, T as in Tom, Y as in yellow, P as in Peter, A as in apple, S as in Sam.
Allison Hartsoe – 45:11 – That’s great and it’s Jodie with an IE as well.
Jodie Antypas – 45:15 – That’s correct. There are not many Antypas� on Linkedin, so you should be able to find me. No problem.
Allison Hartsoe – 45:20 – Excellent. Excellent. Okay, so at the top of the show, he talked about why customer centric research and one of the things you said was that online tools is making it so much faster and very nimble, but you still have the importance of getting a good clear sample and the business strategies we talked about at the end as well, so kind of knowing what you’re going for, having the clear sample to answer that question and then the online tools make your life easier, but the online tools don’t answer the business question for you or inherently give you the right sample. You must know what you’re going for in the first place. We also talked about mobile phones making it possible to do real time research and what I really loved in this section was the, the target audience identification that you’re looking for different audiences and the heterogeneity of the whole audience group when you’re doing customer centric research and that might be a shift in how we thought about research previously when it was really demographically focused.
Allison Hartsoe – 46:25 – Today we’re really looking at the, um, you know, did they stay engaged in the game? We’re not focused so much on just selling the game, but on how do we get them to purchase the next game, which is really building that long-term relationship. So, I really love the angles that you came up with on the customer centric research. And I think it’s, it’s not exactly subtle. I still hear people talk all the time about demographics and it’s not that those aren’t important, but as you illustrate in the examples, the behavioral, attitudinal diversity and motivational diversity, am I right to say that that would be predictive or more predictive than our traditional demographics?
Jodie Antypas – 47:07 – Oh, absolutely. We find much more on motivation and attitudes than we do on traditional demographics.
Allison Hartsoe – 47:14 – That’s what I thought and then we went into three examples. We talked about FIFA and the ethnographic researcher and this was so interesting because you’re looking at the knowledge again of really interesting slice of not just looking at how often do they use it or you know, what geographic area are they from, but you know, is there a level of knowledge low or high, and then we talked about like a flower unfolding and using that as the tip of the spear for research to then find interesting insights behind it and that also led us into a little bit of discussion about avoiding confirmation bias and making sure that your listening for what people are saying this. You also echoed at the end when you talked about the customer isn’t wrong. It’s really the customer telling you their truth. I think by listening to the customer’s truth, you help avoid confirmation bias as well. Would that be true?
Jodie Antypas – 48:13 – Yep. Definitely want to hear what the customer is saying in their own words.
Allison Hartsoe – 48:16 – So going back and talking about battlefield one and what you were talking about, brand essence, this was really looking at the uniqueness of the product and not talking about the fact that it was World War I, but the fact that it was a new experience and epic scale and never the same and just different things that we’re attracting people to the product where World War I was the vehicle or the story behind it, but the experiences was really what the messaging pushed and the iteration that you did to hone that message I think was very, very interesting. Especially the friendship pairs, uh, when you talked about doing qualitative research, uh, and, and needing a great moderator for that. But then also getting the people to come in and have dueling groups or have a, have them in kind of their natural state and pulling them together to get more insights from the messages and the things they say and the things they do all in real time. That seems very, very rich, but it’s not something, again, that you can do by just picking a couple people out of the lineup. You have to very cleanly and clearly picked the people that you want and match them up correctly in order to get that dual group nature. Right?
Jodie Antypas – 49:25 – Yeah. Lots of screening and preparation goes into groups like that.
Allison Hartsoe – 49:29 – I bet. How much screening goes into that? Is it like a 30-minute call or is it, you know, you really deeply digging before you pull them in?
Jodie Antypas – 49:37 – Um, it’s typically a questionnaire and then a follow up interview. And then again, we rescreened people on site before they come in. There’s a lot of people that want to be a professional focus group participant. We don’t want people like that, so we try to trick them and weed them out and make sure they don’t actually know what they’re coming in to talk about.
Allison Hartsoe – 49:55 – Very good. Very good insight there. Cool. Uh, and then we talked finally towards the end about the favorite customer centric research methods, the brand research and the essence, the NPS research, which is always under fire. But I heard the same thing at a, uh, when we, when we talk about CLV and when we talk about that a lot on this show, there’s a really nice connection between CLV, an NPS research and pulling that together to get a sense of satisfaction and value from the customer and that leads right into the core player metrics that you talked about, where are you looking at NPS unique users to get to scale and volume as well as session days, which is unique to electronic arts, but it’s really about engagement. And then of course average spend per player. So, all of these things really focus the research into becoming a powerhouse for your organization. And it sounds to me like by, by getting the organization to be customer centric, you have empowered your researchers to start the right kind of conversations and then bring it through the whole organization because everybody knows what they’re accountable for. Is that fair?
Jodie Antypas – 51:10 – Yep. I think that’s definitely fair. We definitely have shifted the focus more to, to the player and accountability to the player.
Allison Hartsoe – 51:17 – Anything else you’d like to add, Jodie?
Jodie Antypas – 51:19 – Um, I don’t think so. I wish everyone well doing their strategic product research.
Allison Hartsoe – 51:23 – Excellent. Excellent. Well, as always, everything we discussed is at ambitiondata.com/podcast. Jodie, thank you so much for bringing all these insights about research today. I know we’ve just scratched the surface, but you just have such a tremendous well of knowledge in this space and it’s really inspiring to see how one company has pulled it all the way through to make really great action happen within the company, so thank you.
Jodie Antypas – 51:48 – Thanks so much for having me.
Allison Hartsoe – 51:50 – Remember everyone. When you use your data effectively, you can though customer equity, it is not magic. Just a very specific journey that you can follow together. Thank you for joining today’s show. This is your host, Alison Hartsoe, and I have two gifts for you. First, I’ve written a guide for the Customer Centric CMO, which contains some of the best ideas from this podcast, and you can receive it right now. Simply text Ambition Data, one word to 31996, and after you get that white paper, you’ll have the option for the second gift, which is to receive The Signal. Once a month. I put together a list of three to five things I’ve seen that represent customer equity signal not noise, and believe me, there’s a lot of noise out there. Things I include could be smart tools. I’ve run across articles, I’ve shared cool statistics or people and companies I think are making amazing progress as they build customer equity. I hope you enjoy the CMO guide and The Signal. See you next week on the Customer Equity Accelerator.