National Alliance for Physician Competence Discovery Workshop



Norman Johnson

Let's first talk about how I see the challenges you are facing – just to see if we are on the same patient, err, map.

The best suumary of your challenge is “How to evolve an overly-constrained, mature, complex, interdependent system?”  Do you agree?  (most say yes).

The second challenge is that institutional change in your area is happening too slowly. We can look at this from just a host-pathogen perspective: because of world connectivity we are seeing new pathogens at a greater frequency.  For example, the third greatest infectious killer worldwide (HIV) was unknown just 30 years ago. We can also look at this from changes in providers of health care – captured by a quote on the phone the other day in the diversity discussion: “physicians will be put out of business by nurse practitioners.” While it was a joke, is this a concern?  (most say yes)

Another challenge is that you work environment has gone from data poor to data rich to data overload. And it will get worse. (all agree) You are not prepared for the extra information that you will be getting. What are the resources to deal with the extra information that you will be getting?

Finally, the types of patients you see are becoming more diverse, because the world is getting more connected. Today you probably have more culturally diverse people walking into your office (or physician's offices) than you used to.

It may be hard to accept, but the answer to all of the above challenges is diversity – And that’s what we’ll focus on.

A point of trivia that I’ll answer at the end: Which ethnic group is the most genetically diverse on the planet? (many guesses, none right)

This is the second aspect of what I see about where you are headed. It is from one of the concept cards and is a model of evolution in general, but technology specifically. There are three phases or stages: hype or innovation, creating utility, and finally transparency. There is an early phase where things are just forming and there is a lot of hype, then there is a phase that develops utility, and then there is a phase when the resource becomes transparent and it becomes part of your infrastructure – a platform for another cycle of development.

You are still in the hype phase and you are starting to think about developing some utility. What if you do that wrong?

In your endeavor, you are still in the hype phase and you are starting to think about developing utility, but you are far from transparency. What if you do the hype stage wrong?

Norman Johnson

Next slide shows what happened when hype isn’t followed quickly enough by utility.  Think about gene therapy in the mid 80s and how the extreme hype and lack of success greatly reduce subsequent funding and progress? What would have happened if it had been less hyped, and there was more development done before it became marketed? Maybe the progress we are just making now in gene therapy might have happened a decade early?

These thoughts set the stage for the discussion on diversity. We'll talk about diversity and how it plays into those issues. I'm a white male and went to a good school and I can sit in an exit seat - so that doesn't make me the profile of someone that should be talking about diversity. The reason

I became a champion is because of research I did in the mid 90s that surprised me and taught me how important diversity is, but from a very different perspective than from academic biology or ecology. .

How many people listened to Michael Mauboussin's conference call?  Let’s first review what you heard.

You have heard that the utility of experts is being eroded. What do you think about that? There is a study about the pundits and the ones that are on TV come out to be the least accurate about their predictions.

Comment: I once asked Michael Bloomberg about a prediction he made and he said he had to fill the airtime.

Certain systems benefit from expertise – no question, but others do not.  What makes them different?  When expertise fails, what are the options?

Where are computers or expert systems best? Where are experts best? Where are crowds best?

Michael listed from the book “Wisdom of the crowd” the conditions for success in collectives: diversity, an aggregation mechanism and incentives.

He then gave three examples of where collective intelligence (the wisdom of the crowds) can be optimal: 1) Discovering what you know is there, but it’s hidden: he called this the needle in the hay stack problem.  2) The second is the state prediction problem, like guessing the number of jelly beans in the jar. 3) The third is the future prediction problem, like guessing the winners of the academy awards. Michael studied his student’s at Columbia performance and found that the group was usually better than any individual. But that’s not helpful to know when to use what resources.

Michael felt that different types of problems required different types of resources. He divided the problems by: rule based with many options or rule based with limited options or probabilistic with limited options or probabilistic with many options.  And then he backed these up with numbers on accuracy of experts and how much they agreed.

What is insightful is to look at his listing of expert agreement for each of these types of problems. Where there is low expert agreement that there is also low expert accuracy. If you get a lot of different answers to your questions then there is a possibility that other decision making options might be better – particularly collective intelligence.

Someone on the telcon asked the great question: What about a crowd of experts?  We’ll discuss this in a bit, but the answer is that for complex problems, a crowd of experts does worse that a crowd of mixed ability folks.

Here’s a fun example of where “experts” don’t agree: How did we get here? Was it be evolution, intelligent design, or creation? This is something that everyone has a strong opinion about and there are experts in each area.  Again, we’ll see that diversity actually helps answer this question.  Interestingly it appears that the reason for the controversy is that even scientists have a blind side around diversity that makes it difficult to explain the real miracles in nature – like the evolution of the eye (How do cells at a distance coordinate to simultaneously form a lens and imaging?).

There are no easy ways of presenting these ideas, but the best I found is to put it in the context of leadership. In some sense your challenge in the alliance is to provide leadership to the leaders across the system to take them to the next stage. So what are the barriers and what are the resources available to you from a leadership perspective?

My colleague (Jen Watkins) and I did some research on leadership and looked at all the different theories of leadership over the last century - and it's messy. Every time a new idea comes up, it is a variation of another idea and it's not clear where this fits in a leadership landscape.

The insights from leadership studies are the following: initially there was power based leadership – you lead because you had the power to lead.  This was replaced by the idea that you lead because you had unique skills or traits that make you a leader. While not being explicit, this introduced the idea that leadership had something to do with performance. Then the academics went to leadership as collective and shared systems (a version of democracy for organizations). Instead of the leader being the performer, they became enablers.

A major insight of this development is that as our social systems have changed, our theories about leadership have changed, even though many of these mechanisms for leadership where there from the beginning. And many of these new models of leadership were developed to address more complex problem and the faster change being observed.

Observation from group: The variance in leadership in a fish school is 50% from who just happens to be in front and 50% from who has speed and strength – suggesting that part of leadership is accidental. The same is true for flocking of bird: we perceive that there is a leader of the flock, when in actuality the selection is random. Here’s a piece of trivia for you: in some species of fish, if the alpha males dies then the alpha woman will become a male and leads (and functions!) until the new alpha male comes along.  How would you like to live in that world?

leadership continuum
Here is a tool to help you make sense of how leadership is changing and what are your options in your endeavors. It's a landscape for leadership that includes all the past leadership models and the future leadership models.

It is a two dimensional landscape that varies continously – but for simplicity we’ll make it a two-by-two matrix. One axis shows where leadership arises and the other is how leadership arises.

The “Where” refers to how many of the group contributes to the leadership – ranging from one or a few to the entire collective.

The “How” is more subtle – it capture the degree that rules or structure determine the leadership, or in the absence of rules if there is random (like the fish) or opportunistic leadership.  

The lower left box captures most of classic viewpoints of leadership: leaders that use power (rules) or work within the rules, using their skills to become leaders.  Enough said about this box.

The lower right box is what Michael was talking about in the wisdom of the crowds examples: aggregation methods that use collective input.  These leadership resources are captured by many of the online information technology tools recently developed – from prediction market to recommender systems.  They are being used by many companies to make day-to-day decisions and outperform all other forms of leadership;  Marko Rodriguez will talk about these tomorrow.

What about the upper left box where “Leadership emerges with minimal precedence or structure, but still resides in individuals”? We're actually familiar with this box, too – it describes when a hero emerges or when someone that speaks up from the trenches with a great idea that makes them a leader. This is opportunistic leadership without precedence. This is an example of emergent leadership because the leadership really can’t be predicted from knowing all the individual contributions or the rules of the systems – it typically arises outside of the normal leadership structure and from interactions between the parts of the system.

We can make an important observation about emergent leadership relative to classic leadership – that is true for all systems with emergent properties. We have many historical examples of how a hero or unexpected leader becomes part of the system and is supported by the rules and structure (e.g., hero becomes king). This is an example of how an emergent solution can become “non-emergent” and predictable by creating structure to make the solution robust and repeatable. A good example of this is how eBay developed. The founder of eBay, Pierre Omidyar, tells the story of how he had this simple idea about selling things on the web, and then how the social community around eBay developed “emergent” ways to improve the function of the site that wasn’t implemented into the site. All he did was take their emergent ideas and implement them into his system, such as the evaluation system.  He did this again and again, and the rest is history.  The major lessons is that the leadership came from the dynamics of the collective, and he captured their leadership to develop the system.

Another major observation about emergent leadership, but one that is challenging to understand, is that because the emergent leadership can result from the interactions between individuals, the leadership can actually not be associated with specific individuals (embodied), but actually can be disembodied. In some sense the emergent leadership in eBay didn’t reside in the individuals, but emerged from their interactions.  Some of the most recent theories of leadership, such as “adaptive leadership” by Linsky in 2002 captures this concept.  In some sense the leadership in the lower right box is also “disembodied” because it usually comes for information systems (e.g., voting) rather than associated with individuals or even the entire collective. The emergent leadership in the upper row often is disembodied because of its emergent nature – we’ll talk more about this shortly.  While these may be challenging concepts, the disembodied leadership is a reality and must be considered understanding how future leadership resources can be exploited.

The upper right box is the box we want to talk about – What does emergent collective leadership mean? Political scientists would say that the fall of the Berlin Wall was an emergent solution to a long-standing problem – the fall was not planned or even enabled by any leadership. What happened was that there was a loophole in East German laws that allowed small social, non-political gatherings to occur. These gatherings were very diverse: they included soldiers, police, politicians, business people and citizens. And there were many of them all mixing together in these small groups. They collectively began to ask: "why don't we just take down the wall?" And when the emergent leadership concluded that the time had come, because they were so diverse, no part of government could stop them - because "them are us."

Here’s another example. Why can't financial experts outperform the S&P 500? This is especially surprising when you consider that the S&P 500 includes losers as well as winners. A guru in finance made the observation “These are the people who have more knowledge and more training than the vast majority of investors. And yet, neither the superior knowledge nor the superior experience helps them in the long run.” Professional money managers fail to beat the S&P 500 at an average rate of 70% per year. And 90% trail the S&P 500 over a ten-year period. That’s pretty bad. Only two individuals have beat the market every year for more than 10 years in a row.

This is a graph of the utility of experts (taken from Micheal’s book, More Than You Know) as the problem complexity increases. Initially experts have little utility because everyone can perform on simple problems so experts have nothing to offer.  As things get more complex experts provide more utility, but as things get even more complex their utility declines – as in the S&P 500 example above. 

So what about collectives? Michael answered this for hard problems: collectives always outperform the average individual, and often do better than the best.   But if there is an expert that can solve the problem, then the collective is less efficient and has lower utility relative to the expert.

This observation is what the upper right hand box is all about - collectives can outperform where experts begin to drop off.

Why does the collective utility drop off at even higher complexity?

We can do experiments on social insects. How do they find the shortest path to food? Each individual leaves pheromone trails and collectively they find a solution that no single ant discovers. (We note that once the shortest path is found by the collective, then most of the individual ants then use the shortest path. Another example of how an emergent solution become exploited by the system.) The ant foraging example is also a great example of disembodied emergent leadership: The collective knows the shortest path but no individual ant has a job description to find the shortest path – an individual ant can’t even understand what a shorted path is. This afternoon I'll show some simulations to illustrate these ideas. 

The collective knows the shortest path but no individual ant has a job description to find the shortest path.

Here’s another example: the human brain: it had no leader neuron, and the collective neurons (the brain) performs far beyond the performance of an individual neuron.  Truly a miracle – but only because we don’t understand leadership in diverse collectives.   This should make you think twice about what kind of leadership is possible in the upper right box – if we just knew how to harness it. 

So let’s dig into this more.  How can groups solve complex problems that even the individual can’t even understand? How can groups solve hard problems without coordination, without cooperation, and without selection? What will happen when 5 billion people start to use the internet for their own interests and what leadership will emerge when their information starts to interact?  Starts to sound like the human brain, doesn’t it?

To answer these questions let’s look at a simple problem we can understand – the English garden maze – or just a maze.  The maze is a complex problem. When you are in the Maze, you have no perspective to where you are at – you don’t know even if when you turn the next corner if you be at the end, for just lost for a lot longer. When you finally reach the end you have no idea whether if your path is the shortest or even shorter than someone elses.

So what I did was took a maze and had a whole bunch of synthetic people in a computer solve the maze – just like you would – myopic and with a lot of guessing. This is what it looks like.

In the learning stage of the problem you make a random choice at each juncture, unless you started crossing your own path. If solve the maze again using your bread crums (your pheromones), you would improve by eliminating the loops where you came back to the same place – you wouldn’t repeat that unnecessary loop. But your path is likely too long because you don't have the global perspective you don't cut off the incomplete loops you can’t see. Here’s the big ah-ha: A diverse collective has an advantage in closing off the un-closed loops – this is how they find the shortest path.

After I had many individuals independently solve the maze I then combined their information and used the same rule set the individual used on the collective information.  Essentially I had a collective with no more skills than an individual, but they had super information. I then examined the performance of the collective as more individuals were added to the collective. As you see, with just 7 or so individuals you can often achieve the shortest path, and with 20 you converge on the shortest path.

Because this was a computer experiment, I could look at different aspects of the contribution of the information from the individuals and see how changes affected the collective leadership.  I found, for example, that a collective of the best performers are not as good as collective that had lower performers included.  Read that again – it’s a very surprising result.  It occurs because the low performers in their wasted exploration of the maze actually learn a lot about the unconnected paths in the problem and become important to the collective.  So even diversity of performance is important.

We often try to hire the best people. In this situation the emergent path is the best performance, but no one individual has captures performance. If you select on performance only - like for natural selection - you wouldn't get the best answer from the collective. Remember that these upside-down conclusions that challenges our paradigm for a high performing organization are for when problems become complex.  For certain types of problems you have to bring in differences and that leads to a better solution.

What about robustness? Let's take an individual that knows a best path but knows less about other paths. If we introduce noise into his solution and get knocked off his known path, he must rely on wandering to find his best path or find the finish. When a diverse collective gets knocked off their path they have collective information to still optimize the solution and find a short path.  This makes them more robust to noise, uncertainty or mistakes than with the single expert.

Collectives reliably solve problems perfectly that experts can not solve.

Here’s all the conclusions I found in the study.  Collectives realizably solve problems perfectly that experts cannot solve. The emergent solution is not initially embodied in any individual. Diverse collectives are more robust to misinformation.

Because I had a quantitative system, I could ask in detail what metric correlates with performance? I discovered that diversity is what correlates to better performance. There are lots of diversity measures. The best one that I found is diversity defined as uniqueness of information or skills contributed to the collective and not what the collective knew as a group (if everyone in the group knew what the collective knew, then this best diversity measure would be low).

A question: What happens if the individuals interact? Two things happen. You find that they converge faster to the best solution – instead of 7 you only need 4 individuals. But this higher performance with smaller groups is at the expense of lower diversity (they learn together so they tend to have the same information), and therefore they have lower robustness.  

A question: What if end point changes and the optimal solution becomes different? The same conclusions. The expert does very poorly when you move the end point. But the collective does  better because they have a lot of diverse information to adapt to the change.

We’ve talked a lot about ants, what about humans?  The world we live in is different and more complex than for the ants, but the same discoveries apply. In life or even at the job, we each have different goals, but our paths share many common sub-paths where we can find synergy,  even though your goals may be different.  This is the classic water cooler effect, when our social interactions lead to an exchange of information that results in achieving a very hard goal, even if we didn’t appreciate how or how much we contributed.  This makes incentive as an essential requirement in the Wisdom of the Crowds uncertain – what is the incentive in the informal exchange at the water cooler?

Let’s sum up what we talked about.  Consider the Leadership landscape in your problem – it will help you match resources to problems.  Certain types of problems require certain types of resources. In some places you need information system resources, in some places you need experts and

in some cases you need both.  In other places you may need to enable the full diversity of the collective and solve the problems that seem unsolvable.  In general, you improve the quality of your leadership by having more diversity present and you create solutions that are more robust.  In the afternoon talk, we consider the effects of change on leadership and self-organizing systems.

Which ethnic group is the most genetically diverse on the planet? Aboriginals from Australia, because they are the longest stable human sub-species on the planet.  We’ll talk about why this happens in the afternoon talk on Strategies in Ecologies.

Download Norman's Powerpoint slides.

Top of Page