VALIDATE AND MEASURE KPI EFFECTIVENESS IN DESIGN THINKING FOR STARTUPS

This paper discusses the process created for key performance indicators in the Design Thinking Mentoring for startups in the Creative Economy Program created by Samsung. In almost two years, more than twenty Startups were evaluated using this methodology, which allowed to identify the individual evolutions in each stage of the mentoring process and phase of the Design Thinking Process. The evaluation of the indicators gained refinements along this trajectory, allowing to be applied in different segments of startups market, stages of product maturity and business models. The process aims evaluate the evolution of startups from the application of methodology of Design Thinking, being ultimately an evaluation of the method itself.


INTRODUCTION
The value of knowledge for societies and the economic order is widely debated in the literature, and the fuel of modern economy is the knowledge.Implement policies and practices to foment entrepreneurial innovative Startups are not a frivolous occupation.The Samsung Creative Economy Program (SCEP), an unprecedented partnership framed together by the private sector, public sector and the networks of innovation in Brazil and South-Korea, became a national reference for collaboration Startup-Industry and a benchmark for developing knowledge intensive ventures.Nevertheless, the interactions and dialog between Startups and Industries are non-trivial and there are certain barriers.
Then, what design practices would contribute to minimize the distance, sooth the dialog and approach those two antagonist worlds?The advance of User Experience (UX) and User Centered Design (UCD) techniques in the recent years.Faced to this challenge where innovation points a central, the companies suffer with a lack of ability to be creative and meet their customers' needs and expectations.But to create an innovative product that really matters and help people's life, businesses must have the design capacity.They need to fuse design internally within the organization creating a culture that fosters creative thinking.Design is a problem solving discipline and Design Thinking Process, based on the advance of UCD techniques, has been widely applied in the business scenario helping companies to structure the design thoughts and culture, bringing a new way to work and pursue continuous innovation.The most important mindset to create innovation by the Design Thinking Principles is to think more about people and not only in products.It is necessary to observe the scenario as a whole in a combination of understanding people's behaviours, trends immersion, benchmarking as well marketing, sales, and business strategy.The start point is the empathy with the stakeholders: a truly understanding of people's motivation, needs and pain-points to solve, and convert into insights and opportunities that will be explored to create ideas and solutions that really matters.A meaningful innovation put together real user's problems and expectation with new technologies or a new use of a known technology.Many Startups enterprises were based on creative ideas with little focus on design or distorted at middle way -by business alignment, company circums-tances or even due the time.This paper explores the evaluation methodology in Design Thinking application process and metrics to assess individual performance applied to the world of Entrepreneurship and Startups.Discusses its results, benefits, limitations, and contributions for enhancing innovative business models and emerging technologies in the context of Startup-Industry collaboration.The KPIs are applied since from the Boot Camp Program, with an introduction about Design Thinking, until three phases of the mentoring process: (i) Knowing, focused on revisit the users' problems, (ii) Exploring, new ideation session and validation, and (iii) Idea Refinement and Improvement by Service Design Technique.By the application of the KPIs for Design Thinking in the SCEP we have seen the product enhancements validated by own entrepreneurs and startups measurable results, allowing them to anticipate customers changes attending their needs in a more accurate and complete way.

ABOUT SAMSUNG CREATIVE PROGRAM
In April of 2015, the Brazilian Association of Science Parks and Business Incubators (ANPROTEC), the Brazilian subsidiary of Samsung Electronics and the Korean Daegu Center for Creative Economy Innovation (CCEI) signed a Memorandum of Understanding, aiming to disseminate the entrepreneurial Korean Creative Economy model in Brazil (SMART KOREA, 2014), based on the CCEI Daegu accumulated experience and know-how, organized as a collective set of knowledge intensive entrepreneurial resources including, methodologies, technologies, practices and culture, to develop an Acceleration Program designed to seed capital startups.This Acceleration Program is called by Startup Creative Economy and was settled to foment innovation and entrepreneurial activity through the realization of direct investments for creation and development of knowledge intensive Small and Medium Enterprises (SMEs) startups in Brazil.This agreement embraced a wide range of activities, including but not limited to entrepreneurial education, research & development, knowledge and methodology transfer, capacity building, investment and venturing.Its primary objective was to implement a pilot plan, which means the first approach of the Acceleration Program inspired by the South-Korean experience.This pilot plan has been promoted through the Brazilian Government, the private industry and the codified and tacit knowledge transfer from the South Korea to Brazil.The customization process for the development of the SCEP was conducted in obedience to social, techno and economic picture of Brazil and also based on the national methodology of incubation in Brazil, called CERNE (BIZZOTTO et al, 1999).From the Korean model of the Creative Economy and proposed technical review by CERNE's criteria, the SCEP was coined as an acceleration program that aims to develop startups on seed capital stage.The model provides a framework of a startup acceleration process, to be held for nine months, organized into seven stages; namely: (i) Incorporation: the first phase beginning with the startups and incubators incorporation process; (ii) The Boot Camp: the first event dedicated to the preparation of startups and incubators; (iii) The Work Plan: document which details the product vision until the end of the program; (iv) Creative Networking Day (CNDAY): virtual event which focuses on knowledge transfer and spontaneous networking generation; (v) Mentorship & Solutions Ad hoc: understood as complementary measures to mentoring efforts, providing opportunities to startup, for example, access to specialized professionals and special sponsor tools, in order to solve emergency technical problems; (vi) Pitch Day: working event to monitoring preliminary results, held at the end of each month of work; and (vii) Showcase & Graduation: the official closure of the program.For each cycle of acceleration, it is given the term "batch" to a set of attendance startups in the program.In particular, this part of article addresses the: framework of the first batch of the SECP (PROCÓPIO et al, 2016).

ABOUT DESIGN THINKING MENTORING
Borges (2017) said that the Design Thinking is the enhancement of startups innovation, and it is a great collaborative approach between Innovation Areas (IAs), Startups and User Experience Design to promote knowledge through collaboration with Global Value Chains and Innovation Networks.Liedka and Ogilvie (2015) said business is from Mars and design is from Venus, and is possible put them together because, like most opposites, they have a lot to offer each other to complement and enrich the result.But one of the problems pointed by Pease, Dean and Van Bossuyt (2014) regarding Human Centered Design (HCD) approach and related methods is the high costs to implement: "HCD requires large upfront investments by a design team in the form of both time and financial resources in an effort to better understand potential customers."There is no Design Thinking method without a design team and organizations used to prefer fast and small pieces of solutions -even not good enough -than spent money working on quality in a longer time.
Understanding actual Startups scenario -the lack of time, resources and investments -SCEP brings the combination of the two approaches to help companies and enrich their solutions helping them to improve their product's quality.Applying the Design Thinking Process, each company could rethink, explore and enhance their products or services in a very short but intense and productive time.Startups thinking rationality and objectivity.The answers are "right" or "wrong," without half terms.Its decision driver is clear and connected to economic logic, that is accurate and quantifiable, because startups are execution-oriented they usually chase these opportunities in an iterative process: do prototypes, test, do new prototypes, test again and so on.In opposite, Design Thinking Method is behavior and creativity-oriented way to solve problems.By measuring Design Thinking that Royalty and Roth (2015) create an incentive to organizations use the methodology to solve problems.Decisions in design approach are regarded as driven by emotion rather than by logic.Design is seen as a more powerful motivator than motive itself.The answers for design approach are "better" or "worse" (LIEDKA AND OGILVIE, 2015).Design Thinking is a HCD methodology and IDEO defines its process and set of techniques used to create new solutions.It uses design tools to integrate people's needs, technology possibilities and business requirements.It is a methodology basically oriented to solve problems and composed of three stages: (i) Hear Section -when team works to identify a design challenge and gather qualitative data directly from users in widely field work, observing and gathering information to identify areas of opportunity for new product development or the improvement of existing products; (ii) Create Section -when team translates what was learned in the field and proposes concrete solutions to select opportunities identified in the first phase.(iii) Deliver Section -when team moves the solutions and plans into implementation in the field to measure and learn from the implemented designs.It was necessary to adapt the Design Thinking Process due to the context that the startups were inserted and the objectives of the SCEP.The first contact of the startups with Design Thinking in the SCEP was on the event Boot Camp (acceleration process stage ii): there had a talk with introduction about the Design Thinking.This introduction was essential for startups put their users at the center and considering them through the lens of empathy.It was also determinant to them understanding of the process and allowed them to assess which of the phases they believed they would need more focus on mentoring program.(GRIBBONS, 2017) After this first contact with Design Thinking on Boot Camp, the preparation for the SCEP, the Design Thinking Mentoring was planned in three phases of one month each one: Phase I Knowing: Research -This phase is for deepening and exploring, reviewing the original problem in order to understand the users who use or will use the product.In this phase, the most diverse actors (stakeholders) of the process are identified, the project's scope and limits are mapped, providing inputs for the next phase.The applied methods came from Anthropology like Ethnography, observation and field surveys.The goal of this first stage is to review of deeply understanding the scenario and pointing out the needs of the people in relation to the product, bringing reflections, conclusions, insights and problems to be solved.Phase II Exploring: (Re)Definition of Concepts -With the profile of the well-defined target audience (those who will be "served" by the solutions created) the solution is pursued to attend all field data mapped problems and insights, creating innovative ideas (with existing technology or not).At this stage the idea improves, and the possibilities expand.In the mentoring context, the goal was evolving and enhancing the original idea, expanding or even focusing only on a specific solution.At this phase, the startups absorb the insights found and define better their concepts and services.Phase III Idea Improvement: Enhancement and validate the idea -Development of idea with product prototyping, interaction flow and screen execution, tests, adjustments and guidelines specifications to certify that the solution is in accordance with the problem mapped in the first step.

THE KPIS APPLIED IN DESIGN THINKING MENTORING
Royalty, A., & Roth, B. (2016) said that what gets measured gets done, we inspired on this quote since we are working for a big organization, we must analyze the evolution of mentoring and we need to balance the design thinking process, relying on efficiency and productivity to evidence the succeed.
Most organizations that turn to design thinking as a methodology for innovation know the process will be more exploratory and less focused on execution.This means that they need to identity and reward different employee behaviors -more creative behaviors -that support creativi-ty-oriented goals.However, the vast majority of organizations continue to utilize execution-oriented measures without adding any creativity--oriented measures.This results in two major problems.The first is that most organizations lean on metrics, at least in part, to determine the success of new initiatives.If design thinking cannot demonstrate success in a measurable way, it may be abandoned for other innovation methodologies that are easier to track.The second major problem is that employing only execution-oriented measures disincentives the application of design thinking because only behaviors leading to better execution will be rewarded.Previous work has shown that many of these measures simply Hodges-schell and Obrien (2015) said that each stakeholder have a different area to focus on and innovate within, but they also have different ways of measuring success -embodied by their Key Performance Indicators (KPIs).Therefore success depends of stakeholder's engagement on Design Thinking Mentoring, all them are walking together for the success.The key performance indicators drives mentoring and focus on exclusively goals for each phase, when it has a clear goal to achieve, it's easier to keep the focus on what is really important.Following this meaning, we estimated a percentage of completed deliverables per Startup by phases, it was essential for a professional designer to analyze, validate and balance equally all tasks in percentage.The Image 1 represents a example of the deliverables and percentages that were established by the mentor according to analysis and generation of diagnostic through the conversation with the startups and their work plans in Boot Camp (acceleration process stage ii).The work plans were made as self-assessment that each startup planned of its level of maturity within each phase of the Design Thinking Process.Through this it was possible to define the activities that would be executed in each phase and definition of deliverables.This first metric was defined at the beginning of the mentoring process, this metrics of Works Estimated Done by phase and Total Estimated Done by Mentoring are important for progress evaluation and comparison during the program until the end.
Image 1: Self-assessment and defi nition of deliverables.
The Image 2 represents the follow-up performed with the startups, documenting the contacts made and the points aligned.In a visual way it is possible to follow the evolution in the phase of Design Thinking worked through a scale (0) to (100) equivalent to percentage of completed phase.There were startups that delivered, the tasks incomplete or had others reasons, so it was given a second chance to correct or improve their deliverables, in this case, we considered as remade.It was also important through we noted some of them had difficult understanding at the beginning and after that they were more committed.The expectation was that at least 70% of the results were achieved by the startups and below was placed as alert.
Image 2: Follow-up and Evolution in Phase I.
The Image 3 was taken from the evaluation report of the Design Thinking Mentoring.It represents the individual evaluation of each startup during Phase I of the process.These data allow a partial evaluation of the performance and quality of deliverables in the process stage.It is possible to evaluate which startups are in the zone of risk and with the possibility of leaving the program, this evaluation was made in all the three phases.The evaluation was composed with a grade criteria -Committed Goals: Presented cohesion with the objectives proposed (+2), Completed all items of activities proposed (+2), Presented the structured deliverables (+1), Demonstrated evolution (+1) and Self-drive (+1); Deadline: Delivered out of time (0), Delivered up to two days late (+1) and Delivered on time (+2); and Exceeded Expectations (+1) one more point for the startup that had superior commitment.The metric Remade Performance represents the chance to correct or improve their deliverables previously described.The expectation was to reach at least note 7 by the deliverables and below note 5 was placed alert.The Image 4 was taken from the evaluation report of the Design Thinking Mentoring.It represents the individual performance and quality of deliverables ranking of each startup during the three phases of the process and at the end was made an average of these three grades.These data allow an evaluation of which startup performed, that is, they made deliveries with more quality during the program.Each phase grade is an overall performance (average metric in Image 3) about this phase.The Image 6 was taken from the evaluation report of the Design Thinking Mentoring with all metrics as comparative analysis of mentoring results: how advanced the startups were in percentage in Design Thinking before the mentoring (Estimated Done), how they progressed in percentage (Done), the average of grade as quality and performance about the deliverables (Average Performance), and, at the end of the mentoring process, was created one more metric: the Mentoring Knowledge Retention.That metric use criteria scale (-2) to (+2): Did not meet any of the proposed criteria for the mentoring process (-2), Only attended the mentoring, but did not obtain any evolution (-1), Only executed all activities proposed during the mentoring process (0), Had an eff ective participation in the mentoring process (+1) and Had full engagement and eff ort during the mentoring process (+2).This last metric allowed us to evaluate the real engagement of the startups in the program because it allows analysis beyond the deliveries for an eff ectiveness mentoring, it demonstrates the potential of learning and growing each startup.

RESULTS
The Design Thinking Mentoring provided to startups a chance to revisit their original problem, verifying the solution's aff ordance and completeness.To some participants this Human Centered Design approach enabled them to revisit their innovation business model based on this new enhanced or focused idea.All the program mentoring's phases was faced with big challenges that impacted positively for both sides: for the program, it was a great experience for adapting the Design Thinking Methodology and deliverables that fi t in all product stages (idea, Minimum Viable Product -MVP, prototype and product) and for diff erent segments of business.And for startups, the big challenge and conquest was to face many skills and points of view that their couldn't imagine before and as they themselves affirmed, the three months of Design Thinking Mentoring helped them to perceive solution in a holistic way, considering not only their clients, but their users and stakeholders; not only the idea itself, but all touch-points related, all interactions and tasks.Overall, the mentoring help them to identify the solution and its usage as a whole, encompassing the proposition in a more robust and strong one.The participants of this program had a deeply understanding about their users, they had experience of observing people in field, creating personas, developing the user's journey and highlighting real pain points that they couldn't find without this activities.Specially the Service Design they understand that was one of the biggest gains that improved the value of their business because they could see beyond just a product, that their company offers a service, considering that all the faces of interaction and communication within the company results in a structure much larger and more complex.The mentors of this program realize that most of startups, even the more advanced ones, that already had a MVP or a prototype, they have never been direct contact with their users or customers before.Some of them was so focused only about the technology solution that they didn't thought carefully about who was or might could be their target.Some of them had a so big idea and they want to solve many problems that it was almost impossible to set up the MVP.There was also startups that discovered that they don't offer a product but a service to their users and there were so many details between offering a product to their customers.The mentoring had many positives feedbacks from the entrepreneurs.They were surprise because the program helped a lot their entire business from the macro understanding of the solution, activities planning until the details, how to communicate about their purpose, how to interact with their partners and it also helped the product to keep lean and to be more assertive in the development.At the end of the program the startups were ready to maintain the acquired knowledge and culture: keeping close to their users, business indicators, competitors analysis and improving their product or service day by day to keep the innovation on their DNA.These results prove that the design approach associated with the business approach is extremely valuable to enhance startups' solutions and also to reinforce the entrepreneurs' competence.All the program phases and stages were tailored and confirmed as a way to add value to ideas and companies at all.The Samsung Creative Economy Program gains in every cycle, plus experien-ce, pleasure and confidence knowing and validating that it is on the right track through business innovation coupled with design supporting startups and continue to be optimizing existing paths while at the same time ensuring a steady stream of new ideas.Being the factor of change and knowledge in a society, especially in Brazil, is of enormous importance.A program would be possible if we did not look at the structural difficulties and entrepreneurial education flaws for startups.Preparing the new generation of Brazilians was the motivating and guiding us in the darkest hours, and we believe to be our greatest contribution.Transmitting knowledge is the basis of human evolution and we exist as people for this purpose to overcome the impossible and unimaginable barriers.
Renata Zilse: Master graduated designer with User Experience focus.With ergonomics, User Centered Design and Design Thinking background, I always seek the simple and natural experiences for all, in all physical or digital contexts of interaction.At Samsung, I contribute with expertise in methodologies to know the people, their mental models and expectations; planning, creating, applying and analyzing field surveys; activating insights to guide innovative solutions, always with an eye on the user's needs and ease of use.Always sensitizing co--workers to see by the people lens motivating team through the users journeys to catch details that leads to creative and meaningful innovation.Huge expertise in service design, information architecture, natural user interaction and experiences for interactive systems.
Lariane Rossanese : Designer with over 10 years of experience working with technology and innovation, advertising, corporate marketing, fashion, helping clients communicate their key messages and attending people's needs for brands like Samsung, Google, Bayer, C&A, Riachuelo, L'óreal, Movile, Honda, Yamaha, GPTW, Caterpillar, Flight Club, Tokyo Marine and others.A keen and creative problem solver, versatile visual style, eye for detail, delivery high quality interfaces and meaningful experiences that enhance digital projects.Taking a project through from strategy and research to a finished designed and validated with usability tests.
Rodrigo Tobias: With more than 18 years of experience in innovation for leading brands such as Samsung, Google, Sony, Crackle, Disney, Buena Vista International, Bridgestone, Nike and among others.Over the past years I've been focusing on user experience with technology through products & services development, including: mobile, IOT & wearables, no-screen and IA interactions design, conceptual thinking for new products, product design and co-working with fashion & beauty industry.
Edson Arantes: Graduated in Design and working for more than 7 years with digital media, consulting and information technology.He worked in the creative segment for big players like Edenred, Medley, MaryKay, GrupoUltra, WestRock, Emerson, Nestlé and others.Experienced in creating experiences that fuse the principles of design with the development of competencies supported by the application of emerging technologies to adapt to the behaviors and preferences of the users.Currently at Samsung, it supports the structuring of concepts, trends and research for a solid understanding of new ideas focused on people's behaviors and needs.
do not exist yet (ROYALTY; LADENHEIM; ROTH, 2014).Developing a set of usable creativity-oriented measures that capture how well individuals and teams learn and apply design thinking should address these problems.(ADAM ROYALTY & BERNARD ROTH, 2016)

Image 3 :
Performance in Phase I of Design Thinking Mentoring.Source: Internal Report -Analysis Startups Bach#2, own authorship, 2017.
Image 4: Final Ranking Performance.Source: Internal Report -Analysis Startups Bach#2, own authorship, 2017.The Image 5 represents the evolution between what was estimated from the Design Thinking Mentoring in the Boot Camp (Acceleration Process Stage ii) and the level of maturity achieved at the end of the three months of the mentoring process.The data presented in this image is about the Initial Percentage Estimated Done versus Final Percentage Estimated Done in the Design Thinking Mentoring.The expectation and goal for each phase was 70% at least, it was extremely important for internal business reports and decisions point about how committed the startups are with the program.Image 5: Final Evaluation of Progress.Source: Internal Report -Analysis Startups Bach#2, own authorship, 2017.