Due Diligence – Studies Show…

We have just finished a series of three very difficult cases. The common theme in each case involved either a “survey” on trade dress issues or scientific studies supporting the opposition’s claims.

In any matter out before the Federal Courts in the US we have to rely on what is called the Daubert Standard that has superseded the Frye Standard (though some states use the Frye Standard[1]

The Daubert Court[2] suggested that trial judges consider the following factors when making such determination on the merits of the scientific evidence: (1) whether the theory or technique in question can be and has been tested; (2) whether it has been subjected to peer review and publication (publication is not required, "submission to the scrutiny of the scientific community is a component of 'good science,' in part because it increases the likelihood that substantive flaws in methodology will be detected."); (3) its known or potential error rate; (4) the existence and maintenance of standards controlling its operation; and (5) whether it has attracted widespread acceptance within a relevant scientific community. This is a good and reasonable, if sometimes hard to meet standard, especially when dealing with cutting edge issues or issues narrowly focused issues.

In all three matters what I sensed is that something was wrong with the research. Sensing something is wrong is a long way from proving something is wrong. The sense that something is wrong came from in all cases from the definitive nature of the survey and the two scientific studies. It is rare that surveys yield percentages seen in the study or have such definitive conclusions as in the published studies in areas that are too new and too narrow. In all cases counsel agreed with the suspicion and we worked to get at the root of the suspicion. For the survey we requested access to the full methodology and the opportunity to speak with people who took the survey. For the scientific papers we asked for the names of the jurors. In all three cases we were refused. For the survey the survey company claimed some of the methods used were privileged (this raised the judge’s eye). The publishers of the scientific papers considered the paper’s juror’s names and contact information private.

For the study we looked for and found several previous employees. In speaking to the employees we learned that much of the work performed by the expert survey company was outsourced to several smaller firms. With a list of the firms complied from disclosure of the former employees we began calling those firms and asked if they were the firm that did the amazing survey on X. Several shared with us that indeed they had done ‘the work’. It was at this point we subpoenaed the results of the firms as well as all notes, contacts and methodology. What we found was the ‘Expert’ survey firm providing testimony appeared to have reengineered the category of the answers to obtain the results desired by the client. It was like the difference in an appraisal of an item between cost and replacement. The answer can be the same or wildly different depending upon the assumptions.

We had a more piecemeal approach in assessing the prominence of the scientific studies. We searched for those scientists in the fields divined upon and found many had never seen nor heard of the study – even though both had been published in reputable scientific journals. That is when, thanks to a less than subtle hint from one of the alleged jurors we spoke to, we stumbled upon the massive recall of scientific papers by many journals. It seems many papers in obscure disciplines had been submitted to the journals for publication. The submitted papers came with a list of potential jurors complete with contact emails. The emails of the jurors all came back to the person who submitted the paper to be juried. As the story of the scam broke, the journals began to reach out to the jurors by telephone. When the actual people were contacted they had no idea they had juried the paper in question. This lead to more investigations and the recall of hundreds of papers and a large C change in how papers to be published are juried. Unlike cars when recalled and the owners get a notice, no such notice is sent to the readers of the papers, nor do they get vaporized from printed journals. It turned out that one of the papers had been recalled and another paper was by scientists who had other papers recalled.

The opposition argued long and hard to keep the bad science in and I commend them for the effort, but not for their waste of time and money. Engineered results and faked studies have no place in the world of science, business and certainly none in court. It all three cases the judges removed the dodgy science from the proceedings. Two cases failed on the spot and the third, which had rested upon more papers than one – had some other legitimate issues.

Will the companies that spent millions and I do mean millions working with faked science pursue the creators of the fakes? I hope so, but I doubt it.

Science is a process striving toward answers, but it has to be good science otherwise the answers are just well documented lies. Surveys require transparency in all matters and processes. Scientific studies require sufficient sample sizes, outliers should not be discarded but disclosed, they should be well juried, and – here is a big one – repeatable.

For a bit of humour and further reading please look at   http://pdos.csail.mit.edu/scigen/ it is a program that generates random computer research papers – a few of which have been published. Really!

Also my favourite on taking surveys from “Yes Minister” https://www.youtube.com/watch?v=G0ZZJXw4MTA

In short, you may not be able to questions the results but be wary enough to be able to question the process.

[1] The Frye standard, Frye test, or general acceptance test is a test to determine the admissibility of scientific evidence. It provides that expert opinion based on a scientific technique is admissible only where the technique is generally accepted as reliable in the relevant scientific community. In Daubert v Merrell Dow Pharmaceuticals, 509 US 579 (1993), the Supreme Court held that the Federal Rules of Evidence superseded Frye as the standard for admissibility of expert evidence in federal courts.

[2]The Daubert Court refers to a trilogy of cases that came before the US Supreme Court that articulate the Daubert Standard, Daubert v Merrell Dow Pharmaceutical, General Electric Co v Joiner, and Kumho Tire Co.v Carmichael.

L. Burke Files DDP CACM
Mr. Files is an international financial investigator and due diligence expert who has run cases in over 130 countries and has visited over 100 countries. Mr. Files has tackled investigations running from a few hundred thousand dollars to over 20 billion. Along the way, he became familiar with the knowledge of what people need to do, for due diligence, preventing corruption, and to avoid helping criminals launder money. He brings this experience of hands-on investigating and problem-solving experience to his lectures on Due Diligence, AML, and Anti-Corruption. Prior to founding FE&E, Inc., he served as the Director of Corporate Finance for American National an investment bank focused on development stage venture capital. He was also employed by Oppenheimer/Rouse as a commodities specialist trading customer accounts in Agri-Business, 24-hour gold and silver, and foreign currencies. Mr. Files has authored six books, and many white papers and articles. He has been quoted in major publications including The Guardian, The Financial Times, Forbes, US Newsweek, and more. He is the author of the award-winning book Due Diligence For The Financial Professional 2nd Edition. Mr. Files serves on the board of directors for several private companies, funds, and non-profits. The companies include Unicus Research a specialty advisory service for fund managers and family offices, SGS Glazing a specialty glazing design and estimating firm, and NSI a premium spirits company.