Customer feedback enables agility in product development

What is the agile methodology all about and how can real-time customer feedback enable it? Hello and welcome to Affogata’s podcast: Let’s talk customer feedback. We had the pleasure of hosting Eyal Sheinholtz, Senior Director of Product, Experience at eToro.

Coming from a rich technical background as a developer, team leader, and architect, Eyal decided to pivot and dive into the amazing world of Product Management. For the past eight years he has been with eToro, the world's leading social trading and investing network with more than 20M users from 140 countries, as the Senior Director of Product, Experience managing both the Acquisition and Retention aspects of their platform. Eyal loves creating the best experiences for its users from ideation to execution while being data-informed and ensuring product deliveries are of the highest quality.

Did you know that yearly, 30,000 new consumer products are launched? And 95 percent fail. Why? It’s said that businesses over-invest in innovation and under-invest in opportunities to engage with their customers to uncover the first-party data they need.

What you will listen to in this podcast:

1. The link between customers, products, and the companies that create them has never been more important. Eyal shares how crucial is customer feedback for understanding what customers want and need.

2. Eyal touches base on what is the agile methodology in product management and how does gathering and analyzing customer feedback aids this methodology.

3. A conversation on how the product team at eToro filters this feedback to understand if they should go with the customer feedback or not and the impact it has on the product.

4. Eyal shares with us which methods of gathering customer feedback eToro utilizes and the perfect mix between quantitative and qualitative data.

5. Listen until the end of the podcast as Eyal shares one crucial tip for product managers out there who haven't started utilizing customer feedback for more data-driven product decisions.

Audio file

Transcription

Host:

So let's start! For the first question, Eyal I want to start with the most basic question actually, but in general, we hear the term agile a lot nowadays in the product world. Can you tell us a bit about what is the agile methodology in product management and why is it so important in today's product development?

Speaker - Eyal Sheinholtz, Senior Director of Product, Experience at eToro:

Okay, so that's a great question because it's important to understand that in order to be really agile in the organization, the company, the organization, everyone has to be committed to the process. Being agile means that flexibility is a must, which in some areas and some business domains is more acquired. But in general, being flexible has to be aligned within the organization.

And everyone basically needs to understand that changes may and should happen as long as it's communicated. So, for example, defining a roadmap and sticking to it for a full year is kind of the Waterful way to manage it and manage to work within the organization. But if you're truly agile, changes are welcome, especially if it's a very fast-paced landscape like Dory is working in or whatever you're doing. If you are an early-stage startup, even if you are a big organization and a big company, because things change, and if you're agile enough, you can adapt to these changes.

The idea is to set the product strategy or division or the North Star. Everyone calls it whatever they want, and create a roadmap, but also understand and encourage the adaptive approach to the planning and implementation. So you as a company can really respond quickly and adjust to feedback, which we will discuss. But changes in the competitive landscape, changes that you need to do because you decided to pivot, or whatever you need to make the change for being agile is being open to these changes. And again, it's very important for me to stress that it needs to be communicated, so everyone needs to be aligned with it. If we're making a change, do we know what are we making it for? What is the value of making the change? So any of these interrupts should matter to the company and lead us to building products that our users love.

Host:

I actually really like what you mentioned now because I think that transparency now, especially after the pandemic and everything that is going on, is super important in these changes for our company to be flexible. So I think that's actually a great way to describe the Agile method. And how important is customer feedback in the Agile methodology? And how do you integrate the feedback in future sprints in product development?

Speaker - Eyal Sheinholtz, Senior Director of Product, Experience at eToro:

So customer feedback and also internal feedback for that matter is super crucial to actually be agile. So it's not just about the ideas we had when approaching the roadmap of things that we wanted to do. We as a company, we as the product managers that we want to create and we want to build. We all have gut hunches, we all have stakeholders that want to get involved and have their ideas, and we should definitely follow up with them or build whatever we think we should. But building something, releasing it, and continuing to the next one isn't really being agile.

If we don't collect the feedback as early as possible, by the way, like presenting prototypes to users, whether it's like general users or power users doing it through user testing or whatever you want to use, or how do we make sure that we are on the right path? How do we know that we're not investing our resources? It's time it's dev resources, only to just roll back whatever we decided to release. So collecting feedback has to happen in all stages of the product lifecycle from the ideation. And again, there are different approaches to how you can do it. You can have a close group of power users, as I've said, that you can actually discuss it with you can even do it internally. But again, most of the people working for the company actually are in love with the product.

But collecting the feedback and through the development phase, showing these prototypes or even early half-baked features being presented and getting the feedback from the users up until the release is super important, not because it has to change something along the way. You can either, like you asked, have this feedback and add it to future Sprints. It can actually be a show stopper and stop everything because you really fell in love with your idea and decided to go in a specific direction and then took a detour just because of feedback, which is fine, but it should help us build whatever users actually need.

As I said, everything doesn't have to be a showstopper, but you have to take it into consideration. You have to look at the feedback your users give because eventually, they're the users, they are going to use it. So whatever you have in mind can be incredibly misinterpreted by your users. Or we have these times when you look at something and you sit with someone and let them enjoy an unmoderated test of a feature and grab your head and say, “no, this is not what I meant”. But that is what matters. If that user or these users or everyone else doesn't understand what you try to build here, then this feedback is actually super important.

Just rolling out something doesn't mean it will work. So it's basically about balancing it. You can have feedback. You can say, okay, this is a showstopper. You can have feedback. You say, we can interrupt the current process. We can maybe even delay a release just because of feedback or important feedback that we wanted to get, and we got and improved that. And it can also be incorporated into future sprints. It all depends on the way that you give that feedback.

Host:

Right. By the way, when you talk about this balance lately I've been also listening to companies that I agree with. And how do you filter this feedback to understand if you should go with the customer feedback? Or maybe it's not something that you think it will work in the product. Sometimes the customer is not right. How do you know this? How do you create a solution to this? How do you balance it out? It can be hard.

Speaker - Eyal Sheinholtz, Senior Director of Product, Experience at eToro:

Yeah. So I think that it is a challenge. We do get a lot of what's internal feedback from our managers, for example, customer support. And everyone has an opinion and it's fine. They're tied up to their opinion. So we need to make sure that we take all the feedback and prioritize it.

It's not just about how long will it take us to fix something. First of all, take into consideration if we want to fix it if it needs to be fixed. Did I understand what we were trying to do? Maybe a simple thing like adding guidance for something may help improve it so the feedback can be I didn't understand it. Why do we need it? This doesn't work, and it can be just because of just a simple misunderstanding. So it's about taking all the feedback and all the analytics that we have about features and seeing usage, and it's qualitative and quantitative. Whatever feedback we did get from our users and from our tracking tools or whatever platform you're using, we need to make sure that again, some of them are showstoppers. Some of them should be added to future sprints.

We need to make sure that whatever we do decide to take again, I mentioned in the previous answer, it needs to be aligned with everyone and needs to be communicated. So we actually love sharing feedback with our R&D teams because they help us get them on board. So if we do decide to make a change, just before we release something, I want to tell them, I want to be able to share with them. Listen, this is the feedback we got. We actually agree with what they said. We actually agree with what is happening here. We look at together and say, okay, maybe they didn't understand it, maybe we did something wrong and that is fine. So we discussed it as a group, as a team, whether it's just the product managers. And again, as I said, with the team and decide what we want to take. And that is how we prioritize it and make sure that we try to reduce the noise as much as possible because you can have one opinion and 100 great opinions about something in one bad opinion. And maybe that person actually got it and nailed it and said, okay, this is exactly wrong. And we say, although we got 100 good opinions, he's actually right. But again, we do the testing and with opinions, it's not like statistical significance. So we don't actually know. But we try to review each feedback, categorize it, or maybe tag it, and then we can take it. Okay, this is a showstopper, but this is for future sprints. This is actually something that we have to do now, as I've said, and make sure that we on one hand don't actually delay a very big release, and on the other, make sure that we deliver the best experience.

Host:

Yeah. Makes total sense, actually. And I know product managers tend to combine various methods of gathering customer feedback and working on product development. Can you share with us some examples that you find crucial to utilize when developing new products or features?

Speaker - Eyal Sheinholtz, Senior Director of Product, Experience at eToro:

Yeah, sure. So we have various platforms that even if you're not using today, you probably will use. So you can have App store reviews, and you can have NPS surveys that also request feedback. We have at eToro our Social Networks. We have feeds and users are writing. We actually work with our social team which also manages our social accounts like Twitter and Instagram and LinkedIn. And we get a lot of feedback from those social networks as we use Affogata to basically get everything on also our feed.

So with the help of a social team, we don't only inform our users that there is a new feature and then see what they say. We don't monitor it on a daily basis. But these teams, the social team also, by the way, customer support or account managers work directly with our users, know that they can send us the feedback and we can decide okay, this is something that we want to take care of or not and everyone is helping everyone or at least helping us get the relevant feedback.

We work with the different stakeholders to see if maybe we got feedback about a feature that some stakeholders requested and again we can balance it with them to decide, okay, this is important or not. We like to use beta releases or beta testing in which we open specific features, usually the bigger ones, not the small changes for a closed group. We and eToro are lucky enough to have a specific group that is called popular investors which is basically a group over 2000 that are copied a lot. You can copy people in eToro and we tend to open stuff before to them because they're heavily engaged in the product. So they're basically power users. So we get their feedback.

We also run A/B tests whether we inform the user. When I say A/B test, we do have a classic A/B test to see how users react to things as opposed to a control group, but with something use the same mechanism to open specific features in let's say countries or again a close group of users who are not popular investors and get their feedback through surveys. You can use type form, Google forms, or whatever you want to use and then again review the feedback, make sure we understand that maybe have even a few times. We had some Zoom meetings with some of the people who gave us feedback and we wanted to know exactly what they thought just to make sure that we're all on the same page and this is another way to get it.

Host:

I really like it and it's super interesting how you guys at eToro integrate all of this feedback from multiple channels at the end to the kind of provide the best product to the user. So it's really interesting that I think eToro can definitely call itself a customer-obsessed brand. So I know that also quantitative data is extremely important as we have also been discussing now. But sometimes people may forget about the qualitative side. Why do you think this happens and in which way do you think eToro listens to the customer needs? Even though you have discussed it a bit already?

Speaker - Eyal Sheinholtz, Senior Director of Product, Experience at eToro:

I think as I mentioned, quantitative data is really important. We use it a lot. We make sure our features are being used, we make sure to see the conversion rates between different stages, which is fine, but it doesn't tell you exactly how the user feels, it just shows you the user clicked here, did that etcetera. When you have a lot of users, that's totally fine. But we need to look past the data, not overlook it, but look past it. So we need to make sure that we hear what the users have to say. Again, we do it through the following. We have our social team that gets feedback on various social networks. We have the App store views. As I said, we have customer support, the account managers.

We are trying to get exactly what you're saying because again, if you look at data, then you know, okay, a user clicked this and did that, but you don't actually know how it feels. And again, unless you collect it, but most funnels don't actually collect, like the experience. You have something to some customer support and you can read it like a thumbs up, thumbs down. But even if you have that on the usage of specific features, it still won't tell you the entire story. And when we had big releases for some of our redesigned screens, we actually got super valuable feedback, even if it was good and even if it was bad. Good is pretty nice to share. But we also shared the bad feedback that we had, which wasn't a lot. So that's fine. But there were a lot of things that we know there are, like the Aha moment or something dropped. And we understood that, okay, they didn't like it because and some of the things that we did were like very basic and simple improvements just because something wasn't really understood.

And again, we as product managers or as a company needed to understand, and we do that, we know the product very well. And if you look at new users or even users who have been with us for a few years, we know the product the best. And whatever we take for granted is something new for them, and it's something that they might not understand. And the only way to get it isn't through data. It's actually through the feedback where they can write, they communicate it. I like reading feedback when someone says, I was very angry to see that. I was very disappointed when they had their feelings. It gives us a lot more weight to it because I don't want to make my users disappointed or sad, angry or frustrated. I hate to be frustrated. And they are because they can't find something. And maybe you can just add a tool to explain something. It might be something simple and the data won't show you that. It won't show you how they feel.

Host:

Yes. It gives you the why behind the what at the end of the day, yeah. And there must be a lot of differences between the type of feedback you want to collect prelaunch and post-launch of a product or feature. Can you tell us a bit about this? As a product manager, what type of data are you looking to make more informed product decisions in the various stages of a release?

Speaker - Eyal Sheinholtz, Senior Director of Product, Experience at eToro:

I think I mentioned that we try to collect data at all stages of the development. From the idea that we had trying. We usually start with internal feedback. We take it to the different teams, some of them have ideas like the social team. When we show them something, they can say, you know what, there was actually a good discussion about it in the feed. And it's not that we expect them to monitor everything and send us everything. They have their jobs too. But since they are working with their users, so that's the social team, customer support, or account managers, they sometimes speak the voice of the users.

So we try to get whatever we can at the beginning, whether it's sharing design, whether it's just reviewing a PLD. But we do try to get to our users as soon as possible. Again, as I said, with prototypes maybe we did it not long ago with a new screen. We actually showed it to some of the users in a Zoom call and we recorded it with the permission of course, and share it with the rest of the product managers to show them how valuable this kind of feedback can be used.

So we do try to get as much feedback, whether it's internal or external before we write even one line of code. And it doesn't even have to be something active, actually. True story. About two weeks ago we decided to do something this quarter and one of the group leaders actually came to us and said okay, why that? Why is that? And again, I know this Affogata’s podcast, but I actually used Affogata to get no, actually I just searched for the word I don't want to do spoilers if any of our students for that specific thing that we want to develop. And I found a lot of things which aren't like high heavy users using our platform. They actually wrote about it and I got to some of the threads since we use Affogata also for our feed. So I open it in eToro and saw some of the threads and how users bypassed it since we don't have the feature, how they implemented it throughout. And it's amazing.

So I actually started copy-pasting and getting the links to the post and sending it to the team. They are saying, look, this is what our users want. So it's not just about maybe as I said, running it internally or asking users for feedback about something, it's just searching for it. Maybe they already discussed it. And again, some of our users are very engaged, so we get feedback actually from them. And again, we had a very good experience with sharing half-baked designs, trying to show them, okay, this is what we're going with and we usually are very open about it, say okay, sometimes we can make changes, sometimes we can't. But again, as long as we get that feedback and just hold it and say okay, maybe not now, maybe later some of the feedback, yes, we just don't use because maybe a result of not understanding what we wanted to do. But we try to explain and if we see.

It's kind of like an email thread that if you go back and forth too many times, it has to be a meeting. So if we go back and forth in the discussion about it, it's either they didn't understand it or we're building something they really don't want. And we try to stop and say, okay, maybe we need to revisit it. So it's a process, but we try to do it as much as we can before and even post-launch.

So post-launch can be having a survey, as I've said, getting data through the social team where they post like, look at this change and this is how you can see it. And a lot of people comment on that post and we read it. I actually wait every day refresh to see if there are new comments. What are people saying and collecting that feedback there. And that's basically it.

Host:

It's so refreshing to hear you say that. You're so invested in really understanding what customers think. It is really something that I see that it's a trend now more than in the past, really. And it's really nice to hear that actually. So fast pacing companies such as eToro has millions of external data points, external and internal, as you mentioned, from customers. So how do you manage to filter the noise and find the most relevant pieces of information?

Speaker - Eyal Sheinholtz, Senior Director of Product, Experience at eToro:

So I think that's the first part of the real challenge. We have an amazing product analysis team who is a great partner to work with. They help us with the analysis and also raising relevant questions even if we don't come up with them, which is great because that's what makes a great partner. So we work with them trying to understand what we want to do, and what's the right data to look at. I'm putting aside, of course, how do we track stuff and stuff like that? But what do we need to look at?

And the second part is actually, as I said, balancing that feedback and weighing it properly. So, for example, we can have negative feedback from internal stakeholders, but we see that most users like it. So should we change something? What if it's the other way around? So you have something that internally, we love it, but users hate it. Most of the people who will listen to this will probably say, okay, just lose it because users won't like it. That's fine. But some of the things that we do are regulatory requirements. Some of them are asked by internal teams that we need to support. So we should take into consideration all the feedback. We should be able to tag it and prioritize it. And as you said, it can be a lot, but we are trying to. We're not actually just trying. We are managing it properly to make sure that everything is considered, but not everything leads to action. So we don't have to develop something as long as we review and are aware of the feedback that we've got.

Host:

Yeah, makes total sense. And for the last question today Eyal, what would be your number one tip for product managers out there who haven't started utilizing customer feedback for more data-driven product decisions?

Speaker - Eyal Sheinholtz, Senior Director of Product, Experience at eToro:

So I truly believe in good communication across the company, especially for product managers. So I think that the main tip is to collect and communicate the feedback with everyone. Not just the other product managers, not just your VP. It's not just about the different stakeholders or your bosses. I mean, they're not the only people that should understand what users think. It's also our R&D who does the heavy lifting. It's a good example because they should know how impactful the work is. We can share different charts and graphs and funnels showing conversions. That's fine. But seeing the avatar of the user who says this is amazing it looks great. Whatever the feedback is, it really helps.

I mean, you can see how valuable your work is if we take the other way around. If it's negative feedback, as I've said, it usually helps us even push items mid-sprint. So if we do want to make a change going back to being agile, which is fine and we want to make a story out and put a new story in because of feedback, then we have that feedback to show. Listen, this is what users say. And again, we have great teams we work with. They do understand data, they use the platforms we use. But that doesn't replace seeing what a user writes, sharing the screenshot of an App store review, or sending them a link to a post a user wrote on eToro. That's amazing. And if you look at a classic position of a product manager, there's like various drawings of a triangle, we put the product, tech, and business. We as product managers usually represent the voice of the users. So it's on us to make sure they've heard across the organization, not just with the people we work with, and make sure we deliver what they need.

Host:

Eyal, I really loved speaking with you today and I really want to thank you so much for taking the time to participate in today's podcast. First of all, it was super interesting to learn more about the agile methodology, but also your perspective on customer feedback on product management and development.

And that's it for today. Thank you, everybody. And don't forget to visit us at Affogata.com