Technical Competitor Evaluation Guide for AI product managers

Competitor analysis is a crucial aspect of running a successful business, especially in highly competitive markets. By tracking and evaluating competitors, businesses can gain valuable insights into their own market position, anticipate trends, and define their product positioning. It is important to not only have a reactive approach to competitor analysis, but to also proactively consider potential competitors and trends.

In this article, we will delve into the various types of competitor evaluations, as well as different approaches and techniques for conducting a technical competitor analysis. Drawing on my experience as a product manager working on AI products, I will share my insights on the importance and benefits of conducting a thorough competitor analysis.

Competitor analysis, also known as competitive analysis, is the process of researching and examining the products or services of businesses that are attempting to solve similar problems as your own business. This process is often carried out by marketing teams as part of market research or by product managers to identify the strengths and weaknesses of their own products. The scope of competitor analysis can range from specific features to more broad, trend-anticipating research. As a product manager, the scope of my competitor analysis work has often been quite broad and has been directly tied to the goals and scope of the research.

Defining the Scope of AI Competitor Evaluations and Research

Competing companies can be numerous, making it time-consuming and costly to research each one in depth. To make the process more efficient, it is important to define clear research objectives and scope. Research objectives provide the purpose of the research initiative, answering the question “What are we trying to achieve?” Examples of research objectives include defining a new benchmark for current technology or understanding the technical limitations of competitors.

To ensure research objectives are clear and achievable, it is recommended to use the SMART framework. This means making them specific, measurable, achievable, relevant, and time-based. Ensuring they are specific, measurable, and relevant helps maintain focus and motivation. Making sure they are achievable and time-based helps manage resources and stay on schedule.

Defining the scope of AI competitor evaluations and research is crucial in ensuring the project’s success. For example, when studying the technical capabilities of ASR technologies focusing on the “custom vocabularies” feature, the research objectives were specific and relevant, concentrating on understanding the algorithms and techniques used by competitors as well as any technical limitations or challenges they faced. By limiting the scope of the research to a specific time frame, the project could align its efforts with the constraints of the business and stay focused on its goals. This helped to increase the chances of success and ensure that the necessary resources were allocated appropriately. Defining the scope is a key factor in the successful execution of AI competitor evaluations and research.

Conducting AI Competitor Evaluations and Research

Once the scope of the research project has been defined, the next step is to conduct the actual research. This might involve gathering data from a variety of sources, such as company websites, product documentation, and industry publications. It may also involve analyzing this data using tools and techniques such as data visualization or statistical analysis.

There are a few best practices to keep in mind when conducting AI competitor evaluations and research. First, it is important to remain objective and unbiased in the analysis of the data. This means avoiding assumptions or preconceptions about the competitors and focusing on the facts and evidence gathered. It is also important to be thorough and systematic in the data collection and analysis process, in order to ensure that all relevant information is considered.

After defining the scope, objectives, goals, and timelines of the research, you can begin the process of creating a comprehensive list of competitors relevant to your technical competitor evaluations of AI products. It is important to note that this list can include competitors in the larger AI-product industry and not just those within your own specific industry or research. By creating and following through on this list, you can ensure that you have a clear set of guidelines to help direct your research and analysis efforts. Here are the steps that you should take in order to create and use your list of competitors:

1. Identify the relevant competitors in the AI-product industry.
2. Gather and analyze available data on each competitor.
3. Compare and contrast the different competitors in relation to their offerings and capabilities.
4. Utilize the findings to create a well-rounded list of competitors that can be used to guide research and analysis efforts.
5. Investigate and incorporate new competitors as needed to ensure that the list is up-to-date and comprehensive.

By taking these steps, you will be able to create a list of competitors that is both comprehensive and helpful in guiding your research and analysis efforts. This can be an invaluable tool for any AI-product research and analysis project.

Defining evaluation principles, criteria & metrics

At this step, we already have a list of computing technologies, so let’s see how we can define evaluation criteria and metrics for those technologies, on the example of automatic speech recognition (ASR) technology evaluations.

1. Define the evaluation principles, criteria, and metrics: This might include factors such as transcription accuracy, speed, and ease of use. For some of these factors, you might have objective metrics and for others, not.
2. Gather test data: This might include a representative sample of audio recordings in a variety of languages and accents, as well as transcripts of the audio. In the case of Krisp’s technology, I considered only English audios, as in the scope of competitor evaluations, other languages’ performance was not important, as we support only English.
3. Process the test data through the ASR technology: Use the ASR technology to generate transcripts of the audio recordings, and compare the transcripts to the original transcripts to measure the technology’s performance. Here, it is also important to normalize all the transcripts (remove numbers, special characters, etc.), to enable an objective comparison of the outputs.
4. Calculate the evaluation metrics: Use the defined evaluation metrics to measure the performance of the ASR technology. This might include calculating the word error rate (WER) or other metrics such as the percentage of words transcribed correctly.
5. Analyze the results: Compare the performance of the ASR technology to the defined criteria and metrics to determine its strengths and weaknesses.

There are many tools and techniques that can be used in the process of technical analysis and evaluation of competitors. Tools such as Postman, command line, and coding skills can be used to process data through APIs. Excel, Python, and R libraries can be used to analyze, clean, and visualize data. These tools can be invaluable in helping to understand the strengths and weaknesses of competitors, and make informed decisions about one’s own products and strategies.

Communicating the Results of AI Competitor Evaluations and Research

Once the research is complete, it is important to clearly communicate the results to stakeholders within the organization. This might include presenting the findings in a report or presentation format, or summarizing the key points in a memo or email.

Effective communication of the results of AI competitor evaluations and research requires clarity and simplicity. It is important to present the key findings in a straightforward and easily understandable way, highlighting the implications for the business and the actionable steps that can be taken as a result of the research.

Conclusion

AI competitor evaluations and research are essential for businesses looking to stay competitive in the rapidly-evolving world of AI. By defining the scope of the research project and conducting thorough and systematic data collection and analysis, businesses can gain valuable insights into the strengths and weaknesses of their competitors. Clearly communicating the results of the research is also critical in order to ensure that the findings are understood and put into action.