Parachutes are not an argument against RCTs for medical treatments!

Another doctor has recently used parachutes as an example of why some medical treatments don’t need to be tested before using them on patients. That historical claim is wrong.

Arguing for “The search for perfect evidence may be the enemy of good policy,” Greenhalgh, a physician and expert in health care delivery at the University of Oxford, fumed in the Boston Review. “As with parachutes for jumping out of airplanes, it is time to act without waiting for randomized controlled trial evidence.” [emphasis added]….

COVID-19, she argues, has revealed the limits of evidence-based medicine—masks being a potent case in point.

The United Kingdom’s mask crusader
Ellen Ruppel Shell
Science 16 Oct 2020: Vol. 370, Issue 6514, pp. 276-277
DOI: 10.1126/science.370.6514.276

A 2003 article in British Medical Journal claimed after a literature search that “No randomised controlled trials of parachute use have been undertaken,[sic]” and went on to claim that “Individuals who insist that all interventions need to be validated by a randomised controlled trial need to come down to earth with a bump.” This is nonsense. Parachutes were heavily tested by the British air force late in WW I, for example. The issue was controversial at the time because German pilots already had parachutes, and the British military was slow to adopt, perhaps because of NIH (Not Invented Here). Continued trials delayed deployment until after the war was over.

Jet ejection seats, a “super-parachute” invented in the 1940s, received comprehensive engineering tests as various designs were experimented on. Tests ultimately included multiple human trials with volunteers. Despite that, many pilots at the time were hesitant to trust them, but field experience (and lack of alternatives when you were about to crash) led to still-reluctant acceptance. The reluctance stemmed from the dangers of ejection – severe injuries were common, due to high accelerations, collisons with pieces of the aircraft, and so forth. Continued experimentation at many levels (simulations, scale models, dummy pilots, etc.) have led to many improvements over the early designs, and most pilots who eject now are not permanently injured.

Test of a 0/0 ejection by Major Jim Hall, 1965

So parachutes have been, and new designs continue to be, heavily tested. Perhaps the 2003 authors missed them because they did not search obscure engineering and in-house journals written decades before the Internet. What about the “controlled” part of Randomized Controlled Trials? They had not even been invented in 1918; R.A. Fisher’s seminal work on experimental statistics was done in the 1920s and 30s.

More important, engineering trials have something better than randomization: deliberate “corner tests.” With humans and diseases we don’t know all the variables that affect treatment effectiveness, and even if we knew them, we couldn’t measure many of them. But with engineered systems we can figure out most key variables ahead of time. So trials can be run with:

  • Low pilot weight / high pilot weight
  • Low airspeed/high airspeed
  • Low, intermediate, and high altitudes
  • Aircraft at 0 pitch and yaw, all the way to aircraft inverted.
  • Delayed or early seat ejection.
  • Testing prototypes (and now, finite element simulations) can tell us which conditions are most extreme, so not all corners need full-scale tests.

Of course some of these tests will “fail,” e.g. early ejection seats did not work at low altitude and airspeed. Those limits are then written into pilots’ manuals. That is considerably better than we do with many RCT’s, which deliberately choose trial subjects who are more healthy than patients who will ultimately take the medicine.

So let’s stop using this analogy. Parachutes were never adopted without (the equivalent of) RCTs. Thereare many reasons to adopt masks without years of testing, but this is not one of them.

(I have written more about this in my book draft about the evolution of flying from an art to a science.)

Semiconductors get old, and eventually die. It’s getting worse.

I once assumed that semiconductors lasted effectively forever. But even electronic devices wear out. How do semiconductor companies plan for aging?

There has never been a really good solution, and according to this article, problems are getting worse. For example, electronics in cars continue to get more complex (and more safety critical). But cars are used in very different ways after being sold, and in very different climates.This makes it impossible to predict how fast a particular car will age.

Electromigration is one form of aging. Credit:  JoupYoup – Own work, CC BY-SA 4.0, 

When a device is used constantly in a heavy load model for aging, particular stress patterns exaggerate things. An Uber-like vehicle, whether fully automated or not, has a completely different use model than the standard family car that actually stays parked in a particular state a lot of the time, even though the electronics are always somewhat alive. There’s a completely different aging model and you can’t guard-band both cases correctly.

Tesla employees say Gigafactory problems worse than known

By now, Tesla’s manufacturing problems are completely  predictable. See my explanation, after the break. At least Wall St. is starting to catch on.
Also in this article: Tesla’s gigafactory for batteries has very similar problems. That  surprises me; I thought they had competent allies helping with batteries.

But one engineer who works there cautioned that the automated lines still can’t run at full capacity. “There’s no redundancy, so when one thing goes wrong, everything shuts down. And what’s really concerning are the quality issues.”

Source: Tesla employees say Gigafactory problems worse than known

Continue reading

It will be very tricky to test and regulate safety of self-driving cars

My friend Don Norman wrote an op-ed this weekend calling for an FDA-like testing program before autonomous cars are put on the roads in the US. Clearly, some level of government approval is important. But I see lots of problems with using drug testing (FDA = Food and Drug Administration) as a model.

Here is an excerpt from a recent article about testing problems with Uber cars, which were the ones in the recent fatal accident. After the break, my assessment of how to test such cars before they are allowed on American roads.

Waymo, formerly the self-driving car project of Google, said that in tests on roads in California last year, its cars went an average of nearly 5,600 miles before the driver had to take control from the computer to steer out of trouble. As of March, Uber was struggling to meet its target of 13 miles per “intervention” in Arizona, according to 100 pages of company documents obtained by The New York Times and two people familiar with the company’s operations in the Phoenix area but not permitted to speak publicly about it.Yet Uber’s test drivers were being asked to do more — going on solo runs when they had worked in pairs.And there also was pressure to live up to a goal to offer a driverless car service by the end of the year and to impress top executives.

So Uber car performance was more than 100 times worse than Waymo cars?!

Continue reading

Lightweight startup ideas for new entrepreneurship course

I will be co-teaching, with Parand Darugar and Paul Kedrosky, a course on starting a company. This 10 week course requires each team to develop a real project, to the level that they can test it with potential customers. (Beta test, approximately.)

Doing this in 10 weeks can be daunting, but it can be done. I used to teach a course on hardware product development, which required working physical prototypes and 2 long reports in 10 weeks. This year we expect little or no hardware. Instead we are looking for web-enabled, mobile-enabled, or other service ideas with little or no associated hardware. What you will sell is some form of a system that fulfills a need.

Part of entrepreneurship is looking for, and “sensing,” unmet needs. Do this in your everyday life, in your courses, as you watch people on screens. Here are a few ideas to suggest how little is needed. Please, add your own. It’s ok to list an unmet need without saying how to solve it.

Classroom feedback using sheets of paper

A visual barcode based version of the teaching “clicker” that many classrooms use to gather real-time student feedback. Students hold up a piece of cardboard, which the faculty member scans with a phone. Why didn’t I think of that?  Get rid of the student-purchased hardware, and probably increase reliability at the same time. (Although I’m sure it sacrifices some capabilities of the electronic clickers.)plickers2

Search my many sources of free books

My wife and I read a lot of books, still. In the last 5 years a number of access methods for e-books have come along. I can buy ebooks from Apple, Amazon, or others. I can also borrow them free from my local library, Amazon Prime, scribd,  my employer (UCSD’s library) and probably some other services. Each service has its own catalog and its own ways of searching. Checking each of the free services is time consuming, and half the time none of them has what I want. Solve this pain! There are many ways to solve the pain, and the concept could be integrated with existing book services/systems in a variety of ways.


Some of my harder to find papers

Accelerated Learning by Experimentation


In most technologies and most industries, experiments play a central role in organizational learning as a source of knowledge and as a check before changes are implemented. There are four primary types of experiments: controlled, natural, ad-hoc, and evolutionary operation. This paper discusses factors that affect learning by experimentation and how they influence learning rates. In some cases, new ways of experimenting can create an order of magnitude improvement in the rate of learning. On the other hand, some situations are inherently hard to run experiments on, and therefore learning remains slow until basic obstacles are solved. Examples of experimentation are discussed in four domains: product development, manufacturing, consumer marketing, and medical trials.

Keywords: Learning, Experimentation

Full published version:    Bohn Accelerated Learning by Experimentation.    in Learning Curves: Theory, Models, and Applications  edited by Mohamad Y. Jaber, CRC Press, 2011.
Preprint version, through SSRN:

This is a rewritten version of my 1987 paper about manufacturing. Although I never published that paper, it was very influential in the 1990s, including becoming the (plagiarized) framework of a book and several articles by Stefan Thomke.  This book chapter expands and extends the original concepts, including new material from Michael Lapré.