Appirio Testifies Before Congress – Crowdsourcing, Innovation and Prizes

April 10, 2014 Narinder Singh


Yesterday, Appirio had the unique opportunity to testify before the House Subcommittee on Research and Technology in a hearing “Prizes to Spur Innovation and Technology Breakthroughs.”  It’s stunning to see the attention that has suddenly emerged in the last few months on crowdsourcing, open innovation, and prizes / challenges, etc.

Why now?  Well first off, this is not an overnight success.  Topcoder started nearly 13 years ago; xPrize launched their space prize challenge in 2004, Netflix launched their famous million dollar challenge in 2009, and open source communities have disrupted traditional software for nearly two decades.  There are three major reasons we are seeing the sudden rise in attention to this phenomenon:

  • Technology Matters.  A Lot – As Marc Andreesen put it, “Software is eating the world,” every industry is affected.  This is creating a massive crunch on making sure your people, other people, everyone is helping you create the technology that drives your business and its products/services.

  • Because we can – the Internet and Cloud create a global platform.  A derivative to Thomas Friedman’s realization that the world is flat, we are in an age where in 30 seconds in nearly any part of the world an individual developer can get access to the same power / capabilities as the world’s largest companies.  So for the first time it is actually possible to harness the world to help you.

  • Platforms are a go to Business Strategy – It used to be  supply chains competed with one another, now its the platforms that are at the center of extended enterprise strategies.  Your collaboration with customers, developers, partner is a critical one and not something that is built just on signing high level agreements.  Micro connections on deals, with customers (social) and in building and enhancing products can he guided by agreements, but they are executed in real times by people who need low friction ways of connecting and collaborating.

Our written testimony below, talks about several aspects of how communities can be used to drive innovation for the government, highlights key successes, and discusses some of the mechanics of how and why it works.

Video Clip of Testimony

You can also see the entire video testimony here.


Written Testimony of  Narinder Singh

President Topcoder, Co-founder and Chief Strategy Officer, Appirio

Before the Subcommittee on Research and Technology

Hearing – Prizes to Spur Innovation and Technology Breakthroughs

US House of Representatives

April 9, 2014

Chairman, Ranking Member, and Members of the Subcommittee, my name is Narinder Singh, and I am the co-founder and Chief Strategy Officer of Appirio, and the President of it’s Topcoder division. I appreciate the invitation to speak before this esteemed body to address these exciting topics.  We believe crowdsourcing–and specifically community-based development incented by prizes–can lead to dramatic innovation, substantially increase efficiency (reduce cost) and produce much higher quality levels of technology development for both the private and public sectors.

We started Appirio in 2006.  Today, it employs over 900 people with headquarters in San Francisco and Indianapolis.  We have routinely been named on “Best Places to Work” lists in the San Francisco area and in Indiana, have received numerous innovation awards, and in 2012 were named a World Economic Forum Technology Pioneer.

Our Topcoder community consists of over 600,000 designers, developers, and data scientists.  Through Topcoder we have taken hundreds of projects for our clients and crowdsourced them through the community.  This involves breaking down complex projects into smaller components and using prizes – challenge based competitions – to complete them.  As a result we run five to ten thousand competitive challenges (prizes) each year through the Topcoder community.   Our average amount for each of these prizes ranges from a few hundred to few thousand dollars.

This approach has helped leading private sector and government agencies to achieve amazing results in technology that impact science and technology.  In 2013 Harvard Medical School and the February edition of Nature Biotechnology (“Prize-based contests can provide solutions to computational biology problems”) described how the Topcoder community helped Harvard Medical School improve NIH BLAST, an algorithm that aids in genetic research, by 1000x in just two weeks.  The challenge received 122 different submissions and awarded a total of  $6,000 in prizes.

More recently with Topcoder, Appirio helped the research organization of a large pharmaceutical firm improve the performance of software that runs Genome Wide Association Studies (GWAS) – an approach to rapidly scan markers across complete sets of DNA.  We reduced the run-time from approximately ten hours to less than thirty seconds.  This advancement is soon to be shared with the scientific community and will change the way research is conducted. Remarkably, the core of this advancement was driven by a series of less than a dozen contests with approximately $50,000 in prizes.

In government, NASA has partnered with Harvard Business School and Topcoder to create the Harvard NASA Tournament Lab (NTL). This lab focuses on creating cutting edge insights on the optimal design of contests and ways in which the federal government can be more effective in the use of prize-based competitions.  Together we have repeatedly used the concept of connecting a string of smaller prizes together to achieve large scale success.

For example:

  • NASA used a set of challenges to reduce the time for an algorithm to optimize medical supplies on space excursions from three hours to less than thirty seconds (completed).

  • The ISS FIT application, a mobile application to help astronauts track nutrition and health on the International Space Station, was developed through 18 challenges on Topcoder for less than $60,000 in prizes (in final testing).

  • Currently, the NASA Asteroid Grand Challenge series is running on Topcoder with a partnership between NASA and Planetary Resources Inc. (PRI)  to improve the ability to detect asteroids.  It consists of a series of ten interconnected challenges and we hope for a similar leap forward in capability (as we have seen in other challenges) for well under one hundred thousand dollars (recently initiated).

In addition, under NASA’s leadership with their Center of Excellence for Collaborative Innovation (CoECI), the NTL and Topcoder have worked with several other government agencies including the Center of Medicare and Medicaid Services.  A project to modernize CMS infrastructure so that health care providers can easily register (while at the same time states can be proactive in limiting access to bad actors) was developed for the state of Minnesota through a series of challenges. Over eleven months over 140 challenges were executed to complete the project.  The total cost of labor and challenge funds was approximately $1.5 million. Forthcoming research from the NTL indicates that done through traditional models, the initiative would have cost the government nearly $7.5 million.

Predominantly, our business is with private sector companies.  Organizations like Comcast and Ferguson have successfully used Topcoder by breaking large software development projects into smaller competitive prizes. Breaking down large problems into smaller ones offers several key benefits for both projects and prizes.

  • It creates more competitive markets. By breaking down prizes you increase the number of companies or individuals who can compete in the market.  For example there are only a few firms in the world that can compete for a $700M project, but there are tens of thousands of firms that can compete for a $7M one  – and hundreds of millions of individuals who could potentially compete in a $7000 challenge.

  • It de-risks work through specialization.  Crowdsourcing works in part because it allows organizations to tap into a much broader set of talent than otherwise possible.  It also allows organizations to use markets to match the best supply (talent) to the demand of the individual task.  With market volume, community members start to specialize in the areas they are best qualified to deliver outstanding results in.  Quite simply, people can work at what they are best at. When every task in a larger initiative is done in this manner, dramatic results become the norm.

  • It increases participation and education.  While somewhat counter-intuitive, a smaller number of connected prizes can actually increase participation.  Participants in prize challenges certainly compete for the chance to be rewarded.  But with micro competitions, learning is often also a significant objective.  The community of participants shares knowledge extensively and broadly after each competition – creating amongst the best on the job training available.

  • It creates objective measures of skill.  Each competition contributes to a skill rating for the participant.  With a volume of competitions you can objectively see the ratings of an individual and the community improve.  At Topcoder we will often run a series of challenges to create an output (like the ones described above) but also to educate a community on a new technology or product.   As a result, we know our community is better skilled to take on the next set of challenges in that domain.

We believe in large prizes and their capability to create entire markets as proven again and again by my esteemed colleagues from xPrize.  But we also believe micro challenges allow the concept of prizes to be applied to a much larger category of work. By breaking innovative or even common problems down, exponential innovation can occur inside of existing markets.

The rationale for government involvement in prize competitions

We all recognize the importance of scientific and technological advancements in spurring growth.  Traditionally, the government’s role in this has been to subsidize research and development of novel ideas and technologies, in the hope that these will produce fruitful advancements. This remains a critical function.

Note that prize-based competitions constitute a novel complement to this strategy. They differ from this traditional approach in two fundamental ways. First, prizes are paid not based on potential, but rather upon the delivery of results. Practical application and performance is therefore tied to disbursement of funds. Second, they don’t require taking a guess on who will produce the sought-after solution. They leverage competitive forces to reward the individual who delivers best solution. Thus, as demonstrated by the examples above, prizes use government funds judiciously–ensuring practical application and rewarding performance.

Commentary on the FIRST Act as related to prize competitions

We applaud the desire to include prizes and competitions in the FIRST act, but also have concerns around unintended consequences for micro challenges and connected initiatives.

The FIRST Act currently states that even for private sector judges, “All judges shall be required to disclose all personal financial interests.”  It’s not immediately clear if this refers to all interests related to the prize they are administering, or simply all their financial interests.  In the case of the latter, for challenges of a few hundred to a few thousand dollars, it’s an onerous burden and counterproductive to the Act’s intent.

For example, often at Topcoder we use certified, non competing, community members to judge competitions.  In these cases their scorecards and evaluations are open and subject to public peer review.  In other cases, the formal judge may only enforce high level guidelines and settle disputes because the competition itself is judged and evaluated by a computer program that scores each entry.  The intent in the FIRST Act in these regards is appropriate – to create transparency and fairness in oversight.  But the current language could instead hinder adoption of prizes inside the government.

Broader view on making government more tech savvy

More broadly, applying this market-based approach of a large volume of micro-prizes in government requires a reduction in bureaucracy and friction of engagement.  More directly, it would be unlikely Topcoder or similar communities would choose to greatly expand presence in government sectors if requirements of audit and financial reporting remain hardwired for legacy approaches to government contracting.  It’s just too hard to do business with the government.  Even in submitting this written testimony it was required that forty five hard copies be hand delivered in advance – an approach to information sharing that exists only with the government.

All of this in combination can leave firms like mine with few practical options beyond either working with niche groups in the government who are willing to invest significant resources in helping navigate its complicated waters (like NASA);  or through third party firms that specialize predominantly in government work.  Ultimately, rather than government guidelines that attempt to enforce fairness in every scenario, a more competitive market is what will most help government operate more efficiently and be exposed to the tools and techniques that advance innovation.

Creating conditions for success in competitions

Research by Professor Karim Lakhani at Harvard and his colleagues* has shown that contests bring into play three distinct mechanisms:

  1. They enable many independent “shots on goal” so that the likelihood of finding the best solution increases substantially;

  2. Open entry in contests means that people outside of the traditional siloed knowledge domains of the problem can now have the chance to propose unconventional and radical solutions;

  3. People are self-motivated – a variety of reasons drive participation in contests including winning the cash prize, demonstrating expertise to potential employers or their peers or just plain having fun.

All of these mechanisms have been shown to be important to successful contest execution.  For government it is important to pair this with a clear sense of the objective of the prize.  In some cases large cash prizes are used to draw attention to a problem area and spur private investment to advance or even create an industry.  In other cases, many of which have been cited here, prizes provide an opportunity to exponentially advance an area, remove a constraint or just build more effectively.

In all cases it is important to create a clear sets of criteria for participants.  What does it take to ‘win’, how will entries be scored, what timelines will be adhered to, etc.  Idea generation will have very different dynamics than executing on very specific tasks.  Large prizes will have more scrutiny than smaller ones, but the need for clarity and transparency transcend all categories. You will ultimately get what your rules incent.

Conclusion:  the pace of change will not slow down

The rate of technology advancement will not slow down.  Over the past decade we’ve seen our lives transformed into a world of constant connectivity.  In addition to this new global platform of information exchange, we are seeing dramatic advancements in 3D printing, robotics, bio technologies, artificial intelligence, wearable computing and many other domains.  More than ever, we will need the ability to quickly and efficiently tap into the right skills instantaneously.  Even in traditional domains, like creating web based solutions for government, the scale of adoption and need to evolve quickly require a new pace and capability.

The nature of our economic system is built upon the free market and for good reason – in most cases it’s the most efficient system.  Crowdsourcing taps into the power of markets but also democratizes participation beyond select firms to anyone who has the capabilities to contribute.

I thank the Committee for the opportunity to share my perspectives and would be happy to respond to any questions you may have on these topics.

Boudreau, Kevin J., Nicola Lacetera, and Karim R. Lakhani. “Incentives and Problem Uncertainty in Innovation Contests: An Empirical Analysis.” Management Science 57, no. 5 (May 2011): 843–863.

Jeppesen, Lars Bo, and Karim R. Lakhani. “Marginality and Problem-Solving Effectiveness in Broadcast Search.” Organization Science 21 (September–October 2010): 1016–1033.

King, Andrew, and Karim R. Lakhani. “Using Open Innovation to Identify the Best Ideas.” MIT Sloan Management Review 55, no. 1 (Fall 2013): 41–48.

Boudreau, Kevin J., and Karim R. Lakhani. “Using the Crowd as an Innovation Partner.” Harvard Business Review 91, no. 4 (April 2013): 61–69.

Guinan, Eva C., Kevin J. Boudreau, and Karim R. Lakhani. “Experiments in Open Innovation at Harvard Medical School.” Art. 3. MIT Sloan Management Review 54, no. 3 (Spring, 2013): 45–52


Previous Article
5 Custom Salesforce Apps Revealed: How Appirio Uses Software-as-a-Strategic-Weapon
5 Custom Salesforce Apps Revealed: How Appirio Uses Software-as-a-Strategic-Weapon

Appirio has been promoting the idea of Software-as-a-Strategic-Weapon (SaaS-W).  The basic premise of SaaS-...

Next Article
HCM Strategy Discussion: Gamification
HCM Strategy Discussion: Gamification

In their 2013 emerging technology Hype Cycle report, Gartner placed Gamification at the “peak of hype”.  De...