Specified Life: taming data

Showing posts with label taming data. Show all posts

Monday, March 21, 2016

DATA SIMPLIFICATION: Published At Last!

Blog readers can use the discount code: COMP315 for a 30% discount, at checkout.

On March 17, 2016, my book Data Simplification: Taming Information with Open Source Tools was published by Morgan Kaufmann, an imprint of Elsevier. [the Elsevier site indicates that the book is still on preorder, buy you can ignore that]. This past month, I've posted on topics relevant to data simplification. Beginning tomorrow, I'll be moving onto new subjects for this blog site, but I wanted to make one additional comment for anyone who might be on the fence about buying this book.

Most large data projects are total failures (1-21). Furthermore, in my humble opinion, most data projects that are deemed successes at the time of completion are actually failures of a kind, because the data that was collected during the project was abandoned when the project ended. Data shouldn't die. Data should be prepared in a manner that permits anyone (not just the people who planned the project) to confirm the conclusions, to reanalyze the data, to merge the data with other data sources, and to repurpose the data for future projects. To do so, the data must be prepared in a manner that is comprehensible and simplified. My book provides open source tools for creating data that can be used and repurposed, by generations of data scientists.

Enough said! Tomorrow, we move on.

- Jules Berman

key words: computer science, data analysis, data repurposing, data simplification, data science, information science, simplifying data, taming data, jules j berman

References:

[1] Kappelman LA, McKeeman R, Lixuan Zhang L. Early warning signs of IT project failure: the dominant dozen. Information Systems Management 23:31-36, 2006.

[2] Arquilla J. The Pentagon's biggest boondoggles. The New York Times (Opinion Pages) March 12, 2011.

[3] Lohr S. Lessons From Britain's Health Information Technology Fiasco. The New York Times Sept. 27, 2011.

[4] Dismantling the NHS national programme for IT. Department of Health Media Centre Press Release. September 22, 2011. Available from: http://mediacentre.dh.gov.uk/2011/09/22/dismantling-the-nhs-national-programme-for-it/ viewed June 12, 2012.

[5] Whittaker Z. UK's delayed national health IT programme officially scrapped. ZDNet September 22, 2011.

[6] Lohr S. Google to end health records service after it fails to attract users. The New York Times Jun 24, 2011.

[7] An assessment of the impact of the NCI cancer Biomedical Informatics Grid (caBIG). Report of the Board of Scientific Advisors Ad Hoc Working Group, National Cancer Institute, March, 2011.

[8] Heeks R, Mundy D, Salazar A. Why health care information systems succeed or fail. Institute for Development Policy and Management, University of Manchester, June 1999 Available from: http://www.sed.manchester.ac.uk/idpm/research/publications/wp/igovernment/igov_wp09.htm, viewed July 12, 2012.

[9] Brooks FP. No silver bullet: essence and accidents of software engineering. Computer 20:10-19, 1987.

[10] Unreliable research: Trouble at the lab. The Economist October 19, 2013.

[11] Kolata G. Cancer fight: unclear tests for new drug. The New York Times April 19, 2010.

[12] Ioannidis JP. Why most published research findings are false. PLoS Med 2:e124, 2005.

[13] Baker M. Reproducibility crisis: Blame it on the antibodies. Nature 521:274-276, 2015.

[14] Naik G. Scientists' Elusive Goal: Reproducing Study Results. Wall Street Journal December 2, 2011.

[15] Innovation or Stagnation: Challenge and Opportunity on the Critical Path to New Medical Products. U.S. Department of Health and Human Services, Food and Drug Administration, 2004.

[16] Hurley D. Why Are So Few Blockbuster Drugs Invented Today? The New York Times November 13, 2014.

[17] Ioannidis JP. Microarrays and molecular research: noise discovery? The Lancet 365:454-455, 2005.

[18] Vlasic B. Toyota's slow awakening to a deadly problem. The New York Times, February 1, 2010.

[19] Lanier J. The complexity ceiling. In: Brockman J, ed. The next fifty years: science in the first half of the twenty-first century. Vintage, New York, pp 216-229, 2002.

[20] Labos C. It Ain't Necessarily So: Why Much of the Medical Literature Is Wrong. Medscape News and Perspectives. September 09, 2014

[21] Gilbert E, Strohminger N. We found only one-third of published psychology research is reliable - now what? The Conversation. August 27, 2015. Available at: http://theconversation.com/we-found-only-one-third-of-published-psychology-research-is-reliable-now-what-46596, viewed on August 27,2015.

Saturday, March 12, 2016

DATA SIMPLIFICATION: Building Word Lists

Over the next few weeks, I will be writing on topics related to my latest book, Data Simplification: Taming Information With Open Source Tools (release date March 23, 2016). I hope I can convince you that this is a book worth reading.

Blog readers can use the discount code: COMP315 for a 30% discount, at checkout.

Word lists, for just about any written language for which there is an electronic literature, are easy to create. Here is a short Python script, words.py, that prompts the user to enter a line of text. The script drops the line to lowercase, removes the carriage return at the end of the line, parses the result into an alphabetized list, removes duplicate terms from the list, and prints out the list, with one term assigned to each line of output. This words.py script can be easily modified to create word lists from plain-text files (See Glossary item, Metasyntactic variable).

#!/usr/local/bin/python
import sys, re, string
print "Enter a line of text to be parsed into a word list"
line = sys.stdin.readline()
line = string.lower(line)
line = string.rstrip(line)
linearray = sorted(set(re.split(r' +', line)))
for i in range(0, len(linearray)):
   print(linearray[i])
exit

Here is some a sample of output, when the input is the first line of Joyce's Finegans Wake:

c:\ftp>words.py
Enter a line of text to be parsed into a word list

a way a lone a last a loved a long the riverrun, past Eve and Adam's, from 
swerve of shore to bend of bay, brings us by a commodius vicus

a
adam's,
and
bay,
bend
brings
by
commodius
eve
from
last
lone
long
loved
of
past
riverrun,
shore
swerve
the
to
us
vicus
way

Here is a nearly equivalent Perl script, words.pl, that creates a wordlist from a file. In this case, the chosen file happens to be "gettbysu.txt", containing the full-text of the Gettysburg address. We could have included the name of any plain-text file.

#!/usr/local/bin/perl
open(TEXT, "gettysbu.txt");
undef($/); 
$var = lc();
$var =~ s/\n/ /g;
$var =~ s/\'s//g;
$var =~ tr/a-zA-Z\'\- //cd;
@words = sort(split(/ +/, $var));
@words = grep($_ ne $prev && (($prev) = $_), @words);
print (join("\n",@words));
exit;

The words.pl script was designed for speed. You'll notice that it slurps the entire contents of a file into a string variable. If we were dealing with a very large file, that exceeded the functional RAM memory limits of our computer, we would need to modify the script to parse through the file line-by-line.

Aside from word lists you create for yourself, there are a wide variety of specialized knowledge domain nomenclatures that are available to the public (1), (2), (3), (4), (5), (6). Linux distributions often bundle a wordlist, under filename "words", that is useful for parsing and natural language processing applications. A copy of the linux wordlist is available at:

http://www.cs.duke.edu/~ola/ap/linuxwords

Curated lists of terms, either generalized, or restricted to a specific knowledge domain, are indispensable for a variety of applications (e.g., spell-checkers, natural language processors, machine translation, coding by term, indexing. Personally, I have spent an inexcusable amount of time creating my own lists, when no equivalent public domain resource was available.

- Jules Berman

key words: computer science, data analysis, data repurposing, data simplification, data wrangling, information science, simplifying data, taming data, complexity, system calls, Perl, Python, open source tools, utility, word lists, jules j berman

References:

[1] Medical Subject Headings. U.S. National Library of Medicine. Available at: https://www.nlm.nih.gov/mesh/filelist.html, viewed on July 29, 2015.

[2] Berman JJ. A Tool for Sharing Annotated Research Data: the "Category 0" UMLS (Unified Medical Language System) Vocabularies. BMC Med Inform Decis Mak, 3:6, 2003.

[3] Berman JJ Tumor taxonomy for the developmental lineage classification of neoplasms. BMC Cancer 4:88, 2004. http://www.biomedcentral.com/1471-2407/4/88, viewed Jan. 1, 2015.

[4] Hayes CF, O'Connor JC. English-Esperanto Dictionary. Review of Reviews Office, London, 1906. Availalable at: http://www.gutenberg.org/ebooks/16967 viewed on July 29, 2105.

[5] Sioutos N, de Coronado S, Haber MW, Hartel FW, Shaiu WL, Wright LW. NCI Thesaurus: a semantic model integrating cancer-related clinical and molecular information. J Biomed Inform 40:30-43, 2007.

[6] NCI Thesaurus. National Cancer Institute, U.S. National Institutes of Health, Bethesda, MD. Available at: ftp://ftp1.nci.nih.gov/pub/cacore/EVS/NCI_Thesaurus/ viewed on July 29, 2015.

Friday, March 11, 2016

DATA SIMPLIFICATION: ImageMagick

Blog readers can use the discount code: COMP315 for a 30% discount, at checkout.

In yesterday's blog, I discussed using system calls within your scripts. One of my examples called an ImageMagick. Today, I thought I'd describe ImageMagick, and some of its benefits.

ImageMagick is an open source utility that supports a huge selection of robust and sophisticated image editing methods. Its source code download site is:

http://www.imagemagick.org/download/

Users may find it convenient to download the executable binaries, for their specific operating system, from:

http://www.imagemagick.org/script/binary-releases.php

Hundreds of ImageMagick methods are described in detail, and useful examples are provided, at:

http://www.imagemagick.org/Usage/

There are several things you should know about ImageMagick:

1. Unlike the commercial image processing applications, ImageMagick has no graphic user interface. ImageMagick is intended to serve as a command line utility.

2. ImageMagick is powerful. There are hundreds of available methods for creating and modifying images.

3. ImageMagick can be called from Python, Perl, Ruby, and other scripting languages, via system calls or via language-specific ImageMagick interface modules (i.e., PerlMagick, PythonMagick, or RMagick)

Here are a few examples of ImageMagick command lines that can be launched from the system prompt (which happens to be sitting at the c:\ftp subdirectory on my home computer):

Converts an image in SVG format to JPEG format.

c:\ftp>convert mtor_pathway.svg mtor_pathway.jpg

Creates a thumbnail image from image.jpg and converts it to gif format.

c:\ftp>convert -size 600x800 image.jpg -thumbnail 120x160 image_thumb.gif

Applies contrast twice to an image, and produce a new file for the results.

c:\ftp>convert original.jpg -contrast -contrast result.png

Puts the contents of file words.txt into the comment section of tar1.jpg and keeps the same filename for the output file.

c:\ftp>convert -comment @words.txt tar1.jpg tar1.jpg

Displays a verbose description of an image, including the header contents.

c:\ftp>identify -verbose tar1.jpg

The real power of ImageMagick comes when it is inserted into scripts, allowing the programmer to perform batch operations of image collections.

Here is a Python script, image_resize.py, that creates a resized copy of every jpeg file in the current directory.

#!/usr/local/bin/python
import sys, os, re 
filelist = os.listdir(".")
pattern = re.compile(".jpg$")
for filename in filelist:
  if pattern.search(filename):
    out_filename = pattern.sub('_small.jpg', filename)
    print(out_filename)
    cmdstring = "convert " + filename + " -resize 400x267! " + out_filename
    os.system(cmdstring)  
  else:
    continue
exit

Here is a Perl script, factth.pl, that reduces the size of every image in a subdirectory that exclusively contains image files. In this case, the all-image subdirectory is "c:\ftp\factth".

#!/usr/local/bin/perl
$newdir = "c\:\\ftp\\factth";
opendir (MYDIR, $newdir) || die ("Can't open directory");
chdir ($newdir);
while ($file = readdir (MYDIR))
  {
  next if (-d $file);
  next if ($file eq "." || $file eq "..");
  system("convert $file -resize 30% $file");
  }
closedir (MYFILE);
exit;

- Jules Berman

key words: computer science, data analysis, data repurposing, data simplification, data wrangling, information science, simplifying data, taming data, complexity, system calls, Perl, Python, open source tools, utility, Image Magick, ImageMagick, jules j berman

Thursday, March 10, 2016

DATA SIMPLIFICATION: System Calls

Blog readers can use the discount code: COMP315 for a 30% discount, at checkout.

A system call is a command line, inserted into a software program, that interrupts the script while the operating system executes the command line. Immediately afterwords, the script resumes, at the next line. Any utility that runs from the command line can be embedded in any scripting language that supports system calls, and this includes all of the languages discussed in this book.

Here are the properties of system calls that make them useful to programmers:

1. System calls can be inserted into iterative loops (e.g., while loops, for loops), so that they can be repeated any number of times, on collections of files, or data elements.

2. Variables that are generated at run-time (i.e.,during the execution of the script) can be included as arguments added to the system call.

3. The results of the system call can be returned to the script, and used as variables.

4. System calls can utilize any operating system command and any program that would normally be invoked through a command line, including external scripts written in other programming languages. Hence, a system call can initiate an external script written in an alternate programming language, composed at run-time within the original script, using variables generated in the original script, and capturing the output from the external script for use in the original script!

System calls enhance the power of any programming language by providing access to a countless number of external methods and by participating in iterated actions using variables created at run-time.

How does the system call help with the task of data simplification? Data simplification is very often focused on uniformity and reproducibility. If you have 100,000 images, data simplification might involve calling ImageMagick to resize every image to the same height and width. If you need to convert spreadsheet data to a set of triples, than you might need to provide a UUID string (see prior blog) to every triple in the database, all at once. If you are working on a Ruby project, and you need to assert one of Python's numpy methods, on every data file in a large collection of data files, then you might want to create a short Python file that you can be accessed, via a system call, from your Ruby script.

Once you have gotten the hang of including system calls in your scripts, you will probably use them in most of your your data simplification tasks. It's important to know how system calls can be used to great advantage, in Perl, Python, and Ruby. A few examples follow.

The following short Perl script makes a system call, consisting of the DOS "dir" command:

#!/usr/bin/perl
system("dir");
exit;

The "dir" command, launched as a system call, displays the files in the current directory. Here is the equivalent script, in Python:

#!/usr/local/bin/python
import os
os.system("dir")
exit

Notice that system calls in Python require the importation of the os (operating system) module into the script.

Here is an example of a Ruby system call, to ImageMagick's "Identify" utility [note: this only works if you have pre-installed ImageMagick]. The system call instructs the "Identify" utility to provide a verbose description of the image file3320_out.jpg, and to pipe the output into the text file, myimage.txt.

#!/usr/bin/ruby
system("Identify -verbose c:/ftp/3320_out.jpg >myimage.txt")
exit

Here is an example of a Perl system call, to ImageMagick's "convert" utility, that incorporates a Perl variable ($file, in this case) that is passed to the system call [note: this only works if you have pre-installed ImageMagick].

#!/usr/local/bin/perl
$file = "try2.gif";
system("convert -size 350x40 xc:lightgray -font Arial -pointsize 32 -fill black
-gravity north -annotate +0+0 \"Hello, World\" $file");
exit;

The following Python script opens the current directory and parses through every filename, looking for jpeg image files. When a jpeg file is encountered, the script makes a system call to imagemagick, instructing imagemagick's "convert" utility to copy the jpeg file to the thumb drive (designated as the f: drive), in the form of a grayscale image. If you try this script at home, be advised that it requires a mounted thumb drive, in the "f:" drive [note: this only works if you have pre-installed ImageMagick].

#!/usr/local/bin/python
import os, re, string
filelist = os.listdir(".")
for file in filelist:
  if ".jpg" in file:  
    img_in = file
    img_out = "f:/" + file 
    command = "convert " + img_in + " -set colorspace Gray -separate -average " + img_out
    os.system(command)
exit

Let's look at a Ruby script that calls a Perl script, a Python script, and another Ruby script, from within one Ruby script.

Here are the Perl, Python and Ruby scripts that will be called from within a Ruby script:

hi.py
#!/usr/local/bin/python
print("Hi, I'm a Python script")
exit

hi.pl
#!/usr/local/bin/perl
print "Hi, I'm a Perl script\n";
exit;

hi.rb
#!/usr/local/bin/ruby
puts "Hi, I'm a Ruby script"
exit

Here is the Ruby script, call_everyone.rb, that calls external scripts, written in Python, Perl and Ruby:

#!/usr/local/bin/ruby
system("python hi.py")
system("perl hi.pl")
system("ruby hi.rb")
exit

Here is the output of the Ruby script, call_everyone.rb:

c:\ftp>call_everyone.rb
Hi, I'm a Python script
Hi, I'm a Perl script
Hi, I'm a Ruby script

If you have some facility with a variety of language-specific methods and utilities, you can deploy them all from within your favorite scripting language.

- Jules Berman

key words: computer science, data analysis, data repurposing, data simplification, data wrangling, information science, simplifying data, taming data, complexity, system calls, Perl, Python, Ruby, jules j berman

Wednesday, March 9, 2016

DATA SIMPLIFICATION: Specifications to the Rescue!

Blog readers can use the discount code: COMP315 for a 30% discount, at checkout.

Today's blog continues yesterday's discussion of Standards and Specifications

Despite the problems inherent in standards, government committees cling to standards as the best way to share data. The perception is that in the absence of standards, the necessary activities of data sharing, data verification, data analysis, and any meaningful validation of the conclusions will be impossible to achieve (1). This long-held perception may not be true. Data standards, intended to simplify our ability to understand and share data, may have increased the complexity of data science. As each new standard is born, our ability to understand our data seems to diminish. Luckily, many of the problems produced by the proliferation of data standards can be avoided by switching to a data annotation technique broadly known as "specification." Although the terms "specification" and "standard" are used interchangeably, by the incognoscenti, the two terms are quite different from one another. A specification is a formal way of describing data. A standard is a set of requirements, created by an standards development organization, that comprise a pre-determined content and format for a set of data.

A specification is an accepted method for describing objects (physical objects such as nuts and bolts; or symbolic objects, such as numbers; or concepts expresed as text). In general, specifications do not require explicit items of information (i.e. they do not impose restrictions on the content that is included in or excluded from documents), and specifications do not impose any order of appearance of the data contained in the document (i.e., you can mix up and rearrange the data records in a specification if you like). Specifications are not generally certified by a standards organization. Examples of specifications are RDF (Resource Description Framework) produced by the W3C (WorldWide Web Consortium), and TCP/IP (Transfer Control Protocol/Internet Protocol), maintained by the Internet Engineering Task Force. The most widely implemented specifications are simple; thus, easily adopted.

Specifications proved a simple and uniform way of representing the information you choose to include in your reports, messages, and files. Some of the most useful and popular specifications are XML, RDF, Notation 3, and Turtle. In general, specifications do not require explicit items of information (i.e. they do not impose restrictions on the content that is included in or excluded from documents), and specifications do not impose any order of appearance of the data contained in the document (i.e., you can mix up and rearrange the data records in a specification if you like). Specifications are not typically certified by a standards organization, but they are developed by special interest groups. Their legitimacy depends on their popularity.

Files that comply with a specification can be parsed and manipulated by generalized software designed to parse the markup language of the specification (e.g., XML, RDF) and to organize the data into data structures defined within the file.

Specifications serve most of the purposes of a standard, plus providing many important functions that standards typically lack (e.g., full data description, data exchange across diverse types of data sets, data merging, and semantic logic). Data specifications spare us most of the heavy baggage that comes with a standard, which includes: limited flexibility to include changing data objects, locked-in data descriptors, licensing and other intellectual property issues, competition among standards that compete within the same data domain, and bureaucratic overhead (2).

Most importantly, specifications make standards fungible. A good specification can be ported into a data standard, and a reasonably good data standard can be ported into a specification. For example, there are dozens of image formats (e.g., jpeg, png, gif, tiff). Although these formats have not gone through a standards development process, they are used by billions of individuals and have achieved the status as de facto standards. For most of us, the selection of any particular image format is inconsequential. Data scientists have access to robust image software that will convert images from one format to another.

A common mistake committed by data scientists is to convert all their data (legacy data and newly acquired data) into a contemporary standard, and then relying on analytic software that is designed to operate exclusively upon the chosen standard. Doing so only serves to perpetuate their frustrations. You can be certain that your data standard and your software application will be unsuitable for the next generation of data scientists. It makes much more sense to port data into a general specification, from which data can be ported to any current or future data standard.

References:

[1] National Committee on Vital and Health Statistics. Report to the Secretary of the U.S. Department of Health and Human Services on Uniform Data Standards for Patient Medical Record Information. July 6, 2000. Available from: http://www.ncvhs.hhs.gov/hipaa000706.pdf

[2] Berman JJ. Repurposing Legacy Data: Innovative Case Studies. Morgan Kaufmann, Waltham, MA, 2015.

- Jules Berman

key words: computer science, data analysis, data repurposing, data simplification, data wrangling, information science, simplifying data, taming data, complexity, standards, specifications, semantic web, jules j berman

Tuesday, March 8, 2016

DATA SIMPLIFICATION: Substandard Standards

Blog readers can use the discount code: COMP315 for a 30% discount, at checkout.

"The nice thing about standards is that you have so many to choose from." -Andrew S. Tanenbaum

Data standards are the false gods of informatics. They promise miracles, but they can't deliver. The biggest drawback of standards is that they change all the time. If you take the time to read some of the computer literature from the 1970s or 1980s, you will come across the names of standards that have long-since fallen into well-deserved obscurity. You may find that the literature from the 1970s is nearly impossible to read with any level of comprehension, due to the large number of now-obsolete standards-related acronyms scattered through every page. Today's eternal standard is tomorrow's indecipherable gibberish (1).

The Open Systems Interconnection (OSI) was an internet protocol created in 1977 with approval from the International Organization for Standardization. It has been supplanted by TCP/IP, the protocol that everyone uses today. A hand full of programming languages have been recognized as standards by the American National Standards Institute. These include Basic, C,, Ada and Mumps. Basic and C are still popular languages. Ada, recommended by the Federal Government, back in 1995, as the recommended language for all high performance software applications, is virtually forgotten (2). Mumps is still in use, particularly in hospital information systems, but it changed its name to M, lost its allure to a new generation of programmers, and now comes in various implementations that may not strictly conform to the original standard.

In many cases, as a standard matures, it becomes hopelessly complex. As the complexity becomes unmanageable, those who profess to use the standard may develop their own idiosyncratic implementations. Organizations that produce standards seldom provide a mechanism to ensure that the standard is implemented correctly. Standards have long been plagued by non-compliance or (more frequently) under-compliance. Over time, so-called standard-compliant systems tend to become incompatible with one another. The net result is that legacy data, purported to conform to a standard format, is no longer understandable.

Regarding versioning, it is a very good rule of thumb that when you encounter a standard whose name includes a version number (e.g. International Classification of Diseases-10, Diagnostic and Statistical Manual of Mental Disorders-5), you can be certain that the standard is unstable, and must be continually revised. Some continuously revised standards cling tenaciously to life, when they really deserve to die. In some cases, a poor standard is kept alive indefinitely by influential leaders in their fields, or by entities who have an economic stake in perpetuating the standard.

Raymond Kammer, then Director of the U.S. National Institute of Standards and Technology, understood the down-side of standards. In a year 2000 government report, he wrote that "the consequences of standards can be negative. For example, companies-and nations-can use standards to disadvantage competitors. Embodied in national regulations, standards can be crafted to impede export access, sometimes necessitating excessive testing and even redesigns of products. A 1999 survey by the National Association of Manufacturers reported that about half of U.S. small manufacturers find international standards or product certification requirements to be barriers to trade. And according to the Transatlantic Business Dialogue, differing requirements add more than 10% to the cost of car design and development." (3)

As it happens, data standards are seldom, if ever, implemented properly. In some cases, the standards are simply too complex to comprehend. Try as they might, every implementation of a complex standard is somewhat idiosyncratic. Consequently, no two implementations of a complex data standard are equivalent to one another. In many cases, corporations and government agencies will purposefully veer from the standard to accommodate some local exigency. In some cases, a corporation may find it prudent to include non-standard embellishments to a standard to create products or functionalities that cannot be easily reproduced by their competitors. In such cases, customers accustomed to a particular manufacturer's rendition of a standard may find it impossible to switch providers).

The process of developing new standards is costly. Interested parties must send representatives to many meetings. In the case of international standards, meetings occur in locations throughout the planet. Someone must pay for the expertise required to develop the standard, improve drafts, and vet the final version. Standards development agencies become involved in the process, and the end-product must be shepherded through one of the agencies that confer final approval. After a standard is approved, it must be accepted by its intended community of users. Educating a community in the use of a standard is another expense. In some cases, an approved standard never gains traction. Because standards cost a great deal of money to develop, it is only natural that corporate sponsors play a major role in the development and deployment of new standards. Software vendors are clever and have learned to benefit from the standards-making process. In some cases, members of a standards committee may knowingly insert a fragment of their own patented property into the standard. After the standard is released and implemented, in many different vendor systems, the patent holder rises to assert the hidden patent. In this case, all those who implemented the standard may find themselves required to pay a royalty for the use of intellectual property sequestered within the standard (4).

Corporations can profit from standards by obtaining patents on the uses of the standard; not on the patent itself. For example, an open standard may have been created that can be obtained at no cost, and that is popular among its intended users, and that contains no hidden intellectual property. An interested corporation or individual may discover a use for the standard that is non-obvious, novel,and useful; these are the three criteria for awarding patents. The corporation or individual can patent the use of the standard, without needing to patent the standard itself. The patent holder will have the legal right to assert the patent over anyone who uses the standard for the purpose claimed by the patent. This patent protection will apply even when the standard is free and open (4).

Despite the problems inherent in standards, government committees cling to standards as the best way to share data. The perception is that in the absence of standards, the necessary activities of data sharing, data verification, data analysis, and any meaningful validation of the conclusions will be impossible to achieve (5). This long-held perception may not be true. Data standards, intended to simplify our ability to understand and share data, may have increased the complexity of data science. As each new standard is born, our ability to understand our data seems to diminish. Luckily, many of the problems produced by the proliferation of data standards can be avoided by switching to a data annotation technique broadly known as "specification." Although the terms "specification" and "standard" are used interchangeably, by the incognoscenti, the two terms are quite different from one another. A specification is a formal way of describing data. A standard is a set of requirements, created by an standards development organization, that comprise a pre-determined content and format for a set of data.

More on specifications in tomorrow's blog.

References:

[1] Berman JJ. Repurposing Legacy Data: Innovative Case Studies. Morgan Kaufmann, Waltham, MA, 2015.

[2] FIPS PUB 119-1. Supersedes FIPS PUB 119. 1985 November 8. Federal Information Processing Standards Publication 119-1 1995 March 13. Announcing the Standard for ADA. Available from: http://www.itl.nist.gov/fipspubs/fip119-1.htm, viewed August 26, 2012.

[3] Kammer RG. The Role of Standards in Today's Society and in the Future. Statement of Raymond G. Kammer, Director, National Institute of Standards and Technology, Technology Administration, Department of Commerce, Before the House Committee on Science Subcommittee on Technology, September 13, 2000.

[4] Berman JJ. Principles of Big Data: Preparing, Sharing, and Analyzing Complex Information. Morgan Kaufmann, Waltham, MA, 2013.

[5] National Committee on Vital and Health Statistics. Report to the Secretary of the U.S. Department of Health and Human Services on Uniform Data Standards for Patient Medical Record Information. July 6, 2000. Available from: http://www.ncvhs.hhs.gov/hipaa000706.pdf

- Jules Berman

key words: computer science, data analysis, data repurposing, data simplification, data wrangling, information science, simplifying data, taming data, complexity, standards, specificationsjules j berman

Monday, March 7, 2016

DATA SIMPLIFICATION: Poor Identifiers, Horrific Consequences

Blog readers can use the discount code: COMP315 for a 30% discount, at checkout.

All information systems, all databases, and all good collections of data are best envisioned as identifier systems to which data (belonging to the identifier) can be added over time.

If the system is corrupted (e.g., multiple identifiers for the same object, data belonging to one object incorrectly attached to other objects), then the system has no value. You can't trust any of the individual records, and you can't trust any of the analyses performed on collections of records. Furthermore, if the data from a corrupted system is merged with the data from other systems, then all analyses performed on the aggregated data becomes unreliable and useless. This holds true even when every other contributor to the system shares reliable data.

Without proper identifiers, the following may occur: data values can be assigned to the wrong data objects; data objects can be replicated under different identifiers, with each replicant having an incomplete data record (i.e., an incomplete set of data values); the total number of data objects cannot be determined; data sets cannot be verified; and the results of data set analyses will not be valid.

In the past, individuals were identified by their name. When dealing with large numbers of names, it becomes obvious, almost immediately, that personal names are woefully inadequate. In a review of a population of 3.5 million, there occurred nearly 250,000 instances wherein individuals shared the same first and last name; there were 70,000 instances where two people shared the same first name, last name and birthdate (1)!

Aside from the obvious fact that they are not unique (e.g., surnames such as Smith, Zhang, Garcia, Lo, and given names such as John and Susan), one name can have multiple representations. The sources for these variations are many. Here is a partial listing (2):

1. Modifiers to the surname (du Bois, DuBois, Du Bois, Dubois, Laplace, La Place, van de Wilde, Van DeWilde, etc.).

3. Accents that may or may not be transcribed onto records (e.g., acute accent, cedilla, diacritical comma, palatalized mark, hyphen, diphthong, umlaut, circumflex, and a host of obscure markings).

4. Special typographic characters (the combined "ae").

5. Multiple "middle names" for an individual, that may not be transcribed onto records. Individuals who replace their first name with their middle name for common usage, while retaining the first name for legal documents.

6. Latinized and other versions of a single name (Carl Linnaeus, Carl von Linne, Carolus Linnaeus, Carolus a Linne).

7. Hyphenated names that are confused with first and middle names (e.g., Jean-Jacques Rousseau, or Jean Jacques Rousseau; Louis-Victor-Pierre-Raymond, 7th duc de Broglie, or Louis Victor Pierre Raymond Seventh duc deBroglie).

8. Cultural variations in name order that are mistakenly re-arranged when transcribed onto records. Many cultures do not adhere to the Western European name order (e.g., given name, middle name, surname).

9. Name changes; through marriage, legal action, aliasing, pseudonymous posing, or insouciant whim.

I have had numerous conversations with intelligent professionals who are tasked with the responsibility of assigning identifiers to individuals. At some point in every conversation, they will find it necessary to explain that although an individual's name cannot serve as an identifier, the combination of name plus date of birth provides accurate identification in almost every instance. They sometimes get carried away, insisting that the combination of name plus date of birth plus social security number provides perfect identification, as no two people will share all three identifiers: same name, same date of birth, same social security number. This is simply wrong. Let us see what happens when we create identifiers from the name plus birthdate.

Consider this example. Mary Jessica Meagher, born June 7, 1912 decided to open a separate bank account in each of 10 different banks. Some of the banks had application forms, which she filled out accurately. Other banks registered her account through a teller, who asked her a series of questions and immediately transcribed her answers directly into a computer terminal. Ms. Meagher could not see the computer screen and could not review the entries for accuracy.

Here are the entries for her name plus date of birth (1):

1. Marie Jessica Meagher, June 7, 1912 (the teller mistook Marie for Mary).

2. Mary J. Meagher, June 7, 1912 (the form requested a middle initial, not name).

3. Mary Jessica Magher, June 7, 1912 (the teller misspelled the surname).

4. Mary Jessica Meagher, Jan 7, 1912 (the birth month was constrained, on the form, to three letters; Jun, entered on the form, was transcribed as Jan).

5. Mary Jessica Meagher, 6/7/12 (the form provided spaces for the final two digits of the birth year. Through the miracle of bank registration, Mary, born in 1912, was re-born a century later).

6. Mary Jessica Meagher, 7/6/2012 (the form asked for day, month, year, in that order, as is common in Europe).

7. Mary Jessica Meagher, June 1, 1912 (on the form, a 7 was mistaken for a 1).

8. Mary Jessie Meagher, June 7, 1912 (Marie, as a child, was called by the informal form of her middle name, which she provided to the teller).

9. Mary Jessie Meagher, June 7, 1912 (Marie, as a child, was called by the informal form of her middle name, which she provided to the teller, and which the teller entered as the male variant of the name).

10. Marie Jesse Mahrer, 1/1/12 (an underzealous clerk combined all of the mistakes on the form and the computer transcript, and added a new orthographic variant of the surname).

For each of these ten examples, a unique individual (Mary Jessica Meagher) would be assigned a different identifier at each of 10 banks. Had Mary re-registered at one bank, ten times, the results may have been the same.

References:

[1] McCann E. The patient identifier debate: Will a national patient ID system ever materialize? Should it? Healthcare IT News February 18, 2013.

[2] Berman JJ. Principles of Big Data: Preparing, Sharing, and Analyzing Complex Information. Morgan Kaufmann, Waltham, MA, 2013.

- Jules Berman

key words: computer science, data analysis, data repurposing, data simplification, data wrangling, information science, simplifying data, taming data, complexity, identifiers, jules j berman

Sunday, March 6, 2016

Data Simplification: Identifiers

Blog readers can use the discount code: COMP315 for a 30% discount, at checkout.

"I always wanted to be somebody, but now I realize I should have been more specific." -Lily Tomlin

An object identifier is anything associated with the object that persists throughout the life of the object and that is unique to the object (i.e., does not belong to any other object). Everyone is familiar with biometric identifiers, such as fingerprints, iris patterns, and genome sequences. In the case of data objects, the identifier usually refers to a randomly chosen long sequence of numbers and letters that is permanently assigned to the object and which is never assigned to any other data object.

An identifier system is a set of data-related protocols that satisfy the following conditions: 1) Completeness (i.e., every unique object has an identifier); 2) Uniqueness (i.e., each identifier is a unique sequence); 3. Exclusivity (i.e., each identifier is assigned to only one unique object, and to no other object, ever); 4) Authenticity (i.e., objects that receive identification can be verified as the objects that they are intended to be); 5) Aggregation (all information associated with an identifier can be collected); and 6) Permanence (i.e., an identifier is never deleted).

Uniqueness is a very strange concept, especially when applied to the realm of data. For example, if I refer to the number 1, then I am referring to a unique number among other numbers (i.e., there is only one number 1). Yet the number 1 may apply to many different things (i.e., 1 left shoe, 1 umbrella, 1 prime number between 2 and 5). The number 1 makes very little sense to us until we know something about what it measures (e.g., left shoe) and the object to which the measurement applies (e.g., shoe_id_#840354751) (1).

We refer to uniquely-assigned computer-generated character strings as "identifiers". As such, computer-generated identifiers are abstract constructs that do not need to embody any of the natural properties of the object. A long (e.g., 200 character length) character string, consisting of randomly chosen numeric and alphabetic characters is an excellent identifier, because the chances of two individuals being assigned the same string is essentially zero. When we need to establish the uniqueness of some object, such as a shoe or a data record, we bind the object to a contrived identifier.

Jumping ahead just a bit, if we say "part number 32027563 weighs 1 pound," then we are dealing with a meaningful assertion . The assertion tells us three things: 1) that there is a unique thing, known as part number 32027563, 2) the unique thing has a weight, and 3) the weight has a measurement of 1 pound. The phrase "weighs 1 pound" has no meaning until it is associated with a unique object (i.e., part number 32027563 weighs 1 pound). The assertion that "part number 32027563 weighs 1 pound" is a "triple," the embodiment of meaning in the field of computational semantics. A triple consists of a unique, identified object, matched to a pair of data and metadata (i.e., a data element and the description of the data element). Information experts use formal syntax to express triples as data structures.

Returning to the issue of object identification, there are various methods for generating and assigning unique identifiers to data objects (2), (3), (4), (5). Some identification systems assign a group prefix to an identifier sequence that is unique for the members of the group. For example, a prefix for a research institute may be attached to every data object generated within the institute. If the prefix is registered in a public repository, data from the institute can be merged with data from other institutes, and the institutional source of the data object can always be determined. The value of prefixes, and other reserved namespace designations, can be undermined when implemented thoughtlessly (1).

Identifiers are data simplifiers, when implemented properly, because they allow us to collect all of the data associated with a unique object, while ensuring that we exclude that data that should be associated with some other object.

UUID (Universally Unique IDentifier) is an example of one type of algorithm that creates collision-free identifiers that can be generated on command, at the moment when new objects are created (i.e., during the run-time of a software application). Linux systems have a built-in UUID utility, "uuidgen.exe", that can be called from the system prompt.

Here are a few examples of output values generated by the "uuidgen.exe" utility:

$ uuidgen.exe
312e60c9-3d00-4e3f-a013-0d6cb1c9a9fe

$ uuidgen.exe
822df73c-8e54-45b5-9632-e2676d178664

$ uuidgen.exe
8f8633e1-8161-4364-9e98-fdf37205df2f

$ uuidgen.exe
83951b71-1e5e-4c56-bd28-c0c45f52cb8a

$ uuidgen -t
e6325fb6-5c65-11e5-b0e1-0ceee6e0b993

$ uuidgen -r
5d74e36a-4ccb-42f7-9223-84eed03291f9

Data Simplification: Taming Information With Open Source Tools describes simple implementions of UUID utilities under Windows.

Notice that each of the final two examples have a parameter added to the "uuidgen" command (i.e., "-t" and "-r"). There are several versions of the UUID algorithm that are available. The "-t" parameter instructs the utility to produce a UUID based on the time (measured in seconds elapsed since the first second of October 15, 1582, the start of the Gregorian calendar). The "-r" parameter instructs the utility to produce a UUID based on the generation of a pseudorandom number. In any circumstance, the UUID utility produces a fixed length character string suitable as an object identifier. The UUID utility is trusted and widely used by computer scientists.

References:

[1] Berman JJ. Repurposing Legacy Data: Innovative Case Studies. Morgan Kaufmann, Waltham, MA, 2015.

[2] Leach P, Mealling M, Salz R. A Universally Unique IDentifier (UUID) URN Namespace. Network Working Group, Request for Comment 4122, Standards Track. Available from: http://www.ietf.org/rfc/rfc4122.txt, viewed Jan. 1, 2015.

[3] Mealling M. RFC 3061. A URN Namespace of Object Identifiers. Network Working Group, 2001. Available from: https://www.ietf.org/rfc/rfc3061.txt, view Jan. 1, 2015.

[4] Berman JJ. Principles of Big Data: Preparing, Sharing, and Analyzing Complex Information. Morgan Kaufmann, Waltham, MA, 2013.

[5] Berman JJ. Methods in Medical Informatics: Fundamentals of Healthcare Programming in Perl, Python, and Ruby. Chapman and Hall, Boca Raton 2010.

- Jules Berman

key words: computer science, data analysis, data repurposing, data simplification, data wrangling, information science, simplifying data, taming data, complexity, identifiers, jules j berman

Saturday, March 5, 2016

Data Simplification: Hitting the Complexity Barrier

Blog readers can use the discount code: COMP315 for a 30% discount, at checkout.

"Nobody goes there anymore. It's too crowded." -Yogi Berra

It seems that many scientific findings, particularly those findings based on analyses of large and complex data sets, are yielding irreproducible results. We find that we can not depend on the data that we depend on. If you don't believe me, consider these shocking headlines:

1. "Unreliable research: Trouble at the lab." (1) The Economist, in 2013 ran an article examining flawed biomedical research. The magazine article referred to an NIH official who indicated that "researchers would find it hard to reproduce at least three-quarters of all published biomedical findings, the public part of the process seems to have failed." The article described a study conducted at the pharmaceutical company Amgen, wherein 53 landmark studies were repeated. The Amgen scientists were successful at reproducing the results of only 6 of the 53 studies. Another group, at Bayer HealthCare, repeated 63 studies. The Bayer group succeeded in reproducing the results of only one-fourth of the original studies.

2. "A decade of reversal: an analysis of 146 contradicted medical practices." (2) The authors reviewed 363 journal articles, reexamining established standards of medical care. Among these articles were 146 manuscripts (40.2%) claiming that an existing standard of care had no clinical value.

3."Cancer fight: unclear tests for new drug." (3). This New York Times article examined whether a common test performed on breast cancer tissue (Her2) was repeatable. It was shown that for patients who tested positive for Her2, a repeat test indicated that 20% of the original positive assays were actually negative (i.e., falsely positive on the initial test). (3).

4. "Reproducibility crisis: Blame it on the antibodies" (4). Biomarker developers are finding that they cannot rely on different batches of a reagent to react in a consistent manner, from test to test. Hence, laboratory analytic methods, developed using a controlled set of reagents, may not have any diagnostic value when applied by other laboratories, using different sets of the same analytes (4)

5. "Why most published research findings are false." (5). Modern scientists often search for small effect sizes, using a wide range of available analytic techniques, and a flexible interpretation of outcome results. The manuscript's author found that research conclusions are more likely to be false than true (5), (6).

6. "We found only one-third of published psychology research is reliable - now what?" (7). The manuscript authors suggest that the results of a first study should be considered preliminary and tentative. Conclusions have no value until they are independently validated.

Anyone who attempts to stay current in the sciences soon learns that much of the published literature is irreproducible (8); and that almost anything published today might be retracted tomorrow. This appalling truth applies to some of the most respected and trusted laboratories in the world (9), (10), (11), (12), (13), (14), (15), (16). Those of us who have been involved in assessing the rate of progress in disease research are painfully aware of the numerous reports indicating a general slowdown in medical progress (17), (18), (19), (20), (21), (22), (23), (24).

For the optimists, it is tempting to assume that the problems that we may be experiencing today are par for the course, and temporary. It is the nature of science to stall for a while and lurch forwards in sudden fits. Errors and retractions will always be with us so long as humans are involved in the scientific process.

For the pessimists, such as myself, there seems to be something going on that is really new and different; a game changer. This game changer is the "complexity barrier", a term credited to Boris Beizer, who used it to describe the impossibility of managing increasingly complex software products (25). The complexity barrier, known also as the complexity ceiling, reflects the intricacies of Big Science and applies to most of the data analysis efforts undertaken these days (26), (27).

Some of the mistakes that lead to erroneous conclusions in data-intensive research are well-known, and include the following:

1. Errors in sample selection, labeling, and measurement (28), (29), (30). For example, modern biomedical data is high-volume (e.g., gigabytes and larger), heterogeneous (i.e., derived from diverse sources), private (i.e., measured on human subjects), and multi-dimensional (e.g., containing thousands of different measurements for each data record). The complexities of handling such data correctly are daunting (31).

3. Misinterpretation of the data (32), (5), (33), (22), (34), (35), (36), (37)

4. Data hiding and data obfuscation (38), (39)

5. Unverified and unvalidated data (40), (41), (42), (43), (34), (44)

6. Outright fraud (39), (16), (45).

When errors occur in complex data analyses, they are notoriously difficult to discover (40).

Aside from human error, intrinsic properties of complex systems may thwart our best attempts at analysis. For example, when complex systems are perturbed from their normal, steady-state activities, the rules that govern the system's behavior become unpredictable (46). Much of the well-managed complexity of the world is found in machines built with precision parts having known functionality. For example, when an engineer designs a radio, she knows that she can assign names to components, and these components can be relied upon to behave in a manner that is characteristic of its type. A capacitor will behave like a capacitor, and a resistor will behave like a resistor. The engineer need not worry that the capacitor will behave like a semiconductor or an integrated circuit. The engineer knows that the function of a machine's component will never change; but the biologist operates in a world wherein components change their functions, from organism to organism, cell to cell and moment to moment. As an example, cancer researchers discovered an important protein that plays a role in the development of cancer. This protein, p53, was considered to be the primary cellular driver for human malignancy. When p53 mutated, cellular regulation was disrupted, and cells proceeded down a slippery path leading to cancer. In the past few decades, as more information was obtained, cancer researchers have learned that p53 is just one of many proteins that play some role in carcinogenesis, and that the role played by p53 changes depending on the species, tissue type, cellular microenvironment, genetic background of the cell, and many other factors. Under one set of circumstances, p53 may modify DNA repair; under another set of circumstances, p53 may cause cells to arrest the growth cycle (47), (48). It is difficult to predict the biological effect of a protein that changes its primary function based on prevailing cellular conditions.

At the heart of all data analysis is the assumption that systems have a behavior that can be described with a formula or a law, or that can lead to results that are repeatable and to conclusions that can be validated. We are now learning that our assumptions may have been wrong, and that our best efforts at data analysis may be irreproducible.

Science and society may have reached a complexity barrier beyond which nothing can be analyzed and understood with any confidence. In light of the irreproducibility of complex data analyses, it seems prudent to take the follow the following two recommendations:

1. Simplify your complex data, before you attempt analysis.

2. Assume that the first analysis of primary data is tentative and probably wrong. The most important purpose of data analysis is to lay the groundwork for data reanalysis.

References:

[1] Unreliable research: Trouble at the lab. The Economist October 19, 2013.

[2] Prasad V, Vandross A, Toomey C, Cheung M, Rho J, Quinn S, et al. A decade of reversal: an analysis of 146 contradicted medical practices. Mayo Clin Proc 88:790-8, 2013.

[3] Kolata G. Cancer fight: unclear tests for new drug. The New York Times April 19, 2010.

[4] Baker M. Reproducibility crisis: Blame it on the antibodies. Nature 521:274-276, 2015.

[5] Ioannidis JP. Why most published research findings are false. PLoS Med 2:e124, 2005.

[6] Labos C. It Ain't Necessarily So: Why Much of the Medical Literature Is Wrong. Medscape News and Perspectives. September 09, 2014

[7] Gilbert E, Strohminger N. We found only one-third of published psychology research is reliable - now what? The Conversation. August 27, 2015. Available at: http://theconversation.com/we-found-only-one-third-of-published-psychology-research-is-reliable-now-what-46596, viewed on August 27,2015.

[8] Naik G. Scientists' Elusive Goal: Reproducing Study Results. Wall Street Journal December 2, 2011.

[9] Zimmer C. A sharp rise in retractions prompts calls for reform. The New York Times April 16, 2012.

[10] Altman LK. Falsified data found in gene studies. The New York Times October 30, 1996.

[11] Weaver D, Albanese C, Costantini F, Baltimore D. Retraction: altered repertoire of endogenous immunoglobulin gene expression in transgenic mice containing a rearranged mu heavy chain gene. Cell 65:536 (inclusive), 1991.

[12] Chang K. Nobel winner in physiology retracts two papers. The New York Times September 23, 1010.

[13] Fourth paper retracted at Potti's request. The Chronicle March 3, 2011.

[14] Whoriskey P. Doubts about Johns Hopkins research have gone unanswered, scientist says. The Washington Post March 11, 2013.

[15] Lin YY, Kiihl S, Suhail Y, Liu SY, Chou YH, Kuang Z, et al. Retraction: Functional dissection of lysine deacetylases reveals that HDAC1 and p300 regulate AMPK. Nature 482:251-255, retracted November, 2013.

[16] Shafer SL. Letter: To our readers. Anesthesia and Analgesia. February 20, 2009.

[17] Innovation or Stagnation: Challenge and Opportunity on the Critical Path to New Medical Products. U.S. Department of Health and Human Services, Food and Drug Administration, 2004.

[18] Hurley D. Why Are So Few Blockbuster Drugs Invented Today? The New York Times November 13, 2014.

[19] Angell M. The Truth About the Drug Companies. The New York Review of Books Vol 51, July 15, 2004.

[20] Crossing the Quality Chasm: A New Health System for the 21st Century. Quality of Health Care in America Committee, editors. Institute of Medicine, Washington, DC., 2001.

[21] Wurtman RJ, Bettiker RL. The slowing of treatment discovery, 1965-1995. Nat Med 2:5-6, 1996.

[22] Ioannidis JP. Microarrays and molecular research: noise discovery? The Lancet 365:454-455, 2005.

[23] Weigelt B, Reis-Filho JS. Molecular profiling currently offers no more than tumour morphology and basic immunohistochemistry. Breast Cancer Research 12:S5, 2010.

[24] Personalised medicines: hopes and realities. The Royal Society, London, 2005.Available from: https://royalsociety.org/~/media/Royal_Society_Content/policy/publications/2005/9631.pdf, viewed Jan 1, 2015.

[25] Beizer B. Software Testing Techniques. Van Nostrand Reinhold; Hoboken, NJ 2 edition, 1990.

[26] Vlasic B. Toyota's slow awakening to a deadly problem. The New York Times, February 1, 2010.

[27] Lanier J. The complexity ceiling. In: Brockman J, ed. The next fifty years: science in the first half of the twenty-first century. Vintage, New York, pp 216-229, 2002.

[28] Bandelt H, Salas A. Contamination and sample mix-up can best explain some patterns of mtDNA instabilities in buccal cells and oral squamous cell carcinoma. BMC Cancer 9:113, 2009.

[29] Knight, J. Agony for researchers as mix-up forces retraction of ecstasy study. Nature 425:109, September 11, 2003.

[30] Gerlinger M, Rowan AJ, Horswell S, Larkin J, Endesfelder D, Gronroos E, et al. Intratumor heterogeneity and branched evolution revealed by multiregion sequencing. N Engl J Med 366:883-892, 2012.

[31] Berman JJ. Biomedical Informatics. Jones and Bartlett, Sudbury, MA, 2007.

[32] Ioannidis JP. Is molecular profiling ready for use in clinical decision making? The Oncologist 12:301-311, 2007.

[33] Ioannidis JP. Some main problems eroding the credibility and relevance of randomized trials. Bull NYU Hosp Jt Dis 66:135-139, 2008.

[34] Ioannidis JP, Panagiotou OA. Comparison of effect sizes associated with biomarkers reported in highly cited individual articles and in subsequent meta-analyses. JAMA 305:2200-2210, 2011.

[35] Ioannidis JPA, Panagiotou OA. "Comparison of effect sizes associated with biomarkers reported in highly cited individual articles and in subsequent meta-analyses. JAMA 305:2200-2210, 2011.

[36] Ioannidis JP: Excess significance bias in the literature on brain volume abnormalities. Arch Gen Psychiatry 68:773-780, 2011.

[37] Pocock SJ, Collier TJ, Dandreo KJ, deStavola BL, Goldman MB, Kalish LA, et al. Issues in the reporting of epidemiological studies: a survey of recent practice. BMJ 329:883, 2004.

[38] Harris G. Diabetes drug maker hid test data, files indicate. The New York Times July 12, 2010.

[39] Berman JJ. Machiavelli's Laboratory. Amazon Digital Services, Inc., 2010.

[40] Misconduct in science: an array of errors. The Economist. September 10, 2011.

[41] Begley S. In cancer science, many 'discoveries' don't hold up. Reuters Mar 28, 2012,

[42] Abu-Asab MS, Chaouchi M, Alesci S, Galli S, Laassri M, Cheema AK, et al. Biomarkers in the age of omics: time for a systems biology approach. OMICS 15:105-112, 2011.

[43] Moyer VA; on behalf of the U.S. Preventive Services Task Force. Screening for prostate cancer: U.S. Preventive Services Task Force recommendation statement. Ann Intern Med May 21, 2011

[44] How science goes wrong. The Economist Oct 19, 2013.

[45] Martin B. Scientific fraud and the power structure of science. Prometheus 10:83-98, 1992.

[46] Rosen JM, Jordan CT. The increasing complexity of the cancer stem cell paradigm. Science 324:1670-1673, 2009.

[47] Madar S, Goldstein I, Rotter V. Did experimental biology die? Lessons from 30 years of p53 research. Cancer Res 2009;69:6378-6380, 2009.

[48] Zilfou JT, Lowe SW. Tumor Suppressive Functions of p53. Cold Spring Harb Perspect Biol 00:a001883, 2009.

- Jules Berman

key words: computer science, data analysis, data repurposing, data simplification, data wrangling, information science, simplifying data, taming data, complexity, jules j berman

Friday, March 4, 2016

Data Simplification: Chapter Synapses

Over the next few weeks, I will be blogging on topics selected from Data Simplification: Taming Information With Open Source Tools.

Those of you who are computer-oriented know that data analysis typically takes much less time and effort than data preparation. Moreover, if you make a mistake in your data analysis, you can often just repeat the process, using different tools, or a fresh approach to your original question. As long as the data is prepared properly, you and your colleagues can re-analyze your data to your heart's content. Contrariwise, if your data is not prepared in a manner that supports sensible analysis, there's little you can do to extricate yourself from the situation. For this reason, data preparation is, in my experience, much more important than data analysis.

Throughout my career, I've relied on simple open source utilities and short scripts to simplify my data, producing products that were self-explanatory, permanent, and that could be merged with other types of data. Hence, my book.

Blog readers can use the discount code: COMP315 for a 30% discount, at checkout.

Data Simplification: Taming Information With Open Source Tools
Publisher: Morgan Kaufmann; 1 edition (March 23, 2016)
ISBN-10: 0128037814
ISBN-13: 978-0128037812
Paperback: 398 pages
Dimensions: 7.5 x 9.2 inches

Chapter 1, The Simple Life, explores the thesis that complexity is the rate-limiting factor in human development. The greatest advances in human civilization and the most dramatic evolutionary improvements in all living organisms have followed the acquisition of methods that reduce or eliminate complexity.

Chapter 2, Structuring Text, reminds us that most of the data on the Web today is unstructured text, produced by individuals, trying their best to communicate with one another. Data simplification often begins with textual data. This chapter provides readers with tools and strategies for imposing some basic structure on free-text.

Chapter 3, Indexing Text, describes the often undervalued benefits of indexes. An index, aided by proper annotation of data, permits us to understand data in ways that were not anticipated when the original content was collected. With the use of computers, multiple indexes designed for differentpurposes, can be created for a single document or data set. As data accrues, indexes can be updated. When data sets are combined, their respective indexes can be merged. A good way of thinking about indexes is that the document contains all of the complexity; the index contains all of the simplicity. Data scientists who understand how to create and use indexes will be in the best position to search, retrieve, and analyze textual data. Methods are provided for automatically creating customized indexes designed for specific analytic pursuits and for binding index terms to standard nomenclatures.

Chapter 4, Understanding Your Data, describes how data can be quickly assessed, prior to formal quantitative analysis, to develop some insight into what the data means. A few simple visualization tricks and simple statistical descriptors can greatly enhance a data scientist’s understanding of complex and large data sets. Various types of data objects, such as text files, images, and time-series data, can be profiled with a summary signature that captures the key features that contribute to the behavior and content of the data object. Such profiles can be used to find relationships among different data objects, or to determine when data objects are not closely related to one another.

Chapter 5, Identifying and Deidentifying Data, tackles one of the most under-appreciated and least understood issues in data science. Measurements, annotations, properties, and classes of information have no informational meaning unless they are attached to an identifier that distinguishes one data object from all other data objects, and that links together all of the information that has been or will be associated with the identified data object. The method of identification and the selection of objects and classes to be identified relates fundamentally to the organizational model of complex data. If the simplifying step of data identification is ignored or implemented improperly, data cannot be shared, and conclusions drawn from the data cannot be believed. All well-designed information systems are, at their heart, identification systems: ways of naming data objects so that they can be retrieved. Only well-identified data can be usefully deidentified. This chapter discusses methods for identifying data and deidentifying data.

Chapter 6, Giving Meaning to Data, explores the meaning of meaning, as it applies to computer science. We shall learn that data, by itself, has no meaning. It is the job of the data scientist to assign meaning to data, and this is done with data objects, triples, and classifications (see Glossary items, Data object, Triple, Classification, Ontology). Unfortunately, coursework in the information sciences often omits discussion of the critical issue of "data meaning"; advancing from data collection to data analysis without stopping to design data objects whose relationships to other data objects are defined and discoverable. In this chapter, readers will learn how to prepare and classify meaningful data.

Chapter 7, Object-Oriented Data, shows how we can understand data, using a few elegant computational principles. Modern programming languages, particularly object-oriented programming languages, use introspective data (ie, the data with which data objects describe themselves) to modify the execution of a program at run-time; an elegant process known as reflection. Using introspection and reflection, programs can integrate data objects with related data objects. The implementations of introspection, reflection and integration, are among the most important achievements in the field of computer science.

Chapter 8, Problem Simplification, demonstrates that it is just as important to simplify problems as it is to simplify data. This final chapter provides simple but powerful methods for analyzing data, without resorting to advanced mathematical techniques. The use of random number generators to simulate the behavior of systems, and the application of Monte Carlo, resampling, and permutative methods to a wide variety of common problems in data analysis, will be discussed. The importance of data reanalysis, following preliminary analysis, is emphasized.

TABLE OF CONTENTS

Chapter 0. Preface
   References for Preface
   Glossary for Preface

Chapter 1. The Simple Life
   Section 1.1. Simplification drives scientific progress
   Section 1.2. The human mind is a simplifying machine
   Section 1.3. Simplification in Nature
   Section 1.4. The Complexity Barrier
   Section 1.5. Getting ready
   Open Source Tools for Chapter 1
      Perl
      Python
      Ruby
      Text Editors
      OpenOffice
      Command line utilities
      Cygwin, Linux emulation for Windows
      DOS batch scripts
      Linux bash scripts
      Interactive line interpreters
      Package installers
      System calls
References for Chapter 1
   Glossary for Chapter 1

Chapter 2. Structuring Text
   Section 2.1. The Meaninglessness of free text
   Section 2.2. Sorting text, the impossible dream
   Section 2.3. Sentence Parsing
   Section 2.4. Abbreviations
   Section 2.5. Annotation and the simple science of metadata
   Section 2.6. Specifications Good, Standards Bad
   Open Source Tools for Chapter 2
      ASCII
      Regular expressions
      Format commands
      Converting non-printable files to plain-text
      Dublin Core
   References for Chapter 2
   Glossary for Chapter 2

Chapter 3. Indexing Text
   Section 3.1. How Data Scientists Use Indexes
   Section 3.2. Concordances and Indexed Lists
   Section 3.3. Term Extraction and Simple Indexes
   Section 3.4. Autoencoding and Indexing with Nomenclatures
   Section 3.5. Computational Operations on Indexes
   Open Source Tools for Chapter 3
      Word lists
      Doublet lists
      Ngram lists
   References for Chapter 3
   Glossary for Chapter 3

Chapter 4. Understanding Your Data
   Section 4.1. Ranges and Outliers
   Section 4.2. Simple Statistical Descriptors
   Section 4.3. Retrieving Image Information
   Section 4.4. Data Profiling
   Section 4.5. Reducing data
   Open Source Tools for Chapter 4
      Gnuplot
      MatPlotLib
      R, for statistical programming
      Numpy
      Scipy
      ImageMagick
      Displaying equations in LaTex
      Normalized compression distance
      Pearson's correlation
      The ridiculously simple dot product
   References for Chapter 4 
   Glossary for Chapter 4

Chapter 5. Identifying and Deidentifying Data
   Section 5.1. Unique Identifiers
   Section 5.2. Poor Identifiers, Horrific Consequences
   Section 5.3. Deidentifiers and Reidentifiers
   Section 5.4. Data Scrubbing
   Section 5.5. Data Encryption and Authentication
   Section 5.6. Timestamps, Signatures, and Event Identifiers
   Open Source Tools for Chapter 5
      Pseudorandom number generators
      UUID
      Encryption and decryption with OpenSSL
      One-way hash implementations
      Steganography
   References for Chapter 5
   Glossary for Chapter 5

Chapter 6. Giving Meaning to Data
   Section 6.1. Meaning and Triples
   Section 6.2. Driving Down Complexity with Classifications
   Section 6.3. Driving Up Complexity with Ontologies
   Section 6.4. The unreasonable effectiveness of classifications
   Section 6.5. Properties that Cross Multiple Classes
   Open Source Tools for Chapter 6
      Syntax for triples
      RDF Schema
      RDF parsers
      Visualizing class relationships
   References for Chapter 6
   Glossary for Chapter 6

Chapter 7. Object-oriented data
   Section 7.1. The Importance of Self-explaining Data
   Section 7.2. Introspection and Reflection
   Section 7.3. Object-Oriented Data Objects
   Section 7.4. Working with Object-Oriented Data
   Open Source Tools for Chapter 7
      Persistent data
      SQLite databases
   References for Chapter 7
   Glossary for Chapter 7

Chapter 8. Problem simplification
   Section 8.1. Random numbers
   Section 8.2. Monte Carlo Simulations
   Section 8.3. Resampling and Permutating
   Section 8.4. Verification, Validation, and Reanalysis
   Section 8.5. Data Permanence and Data Immutability
   Open Source Tools for Chapter 8
      Burrows Wheeler transform
      Winnowing and chaffing
   References for Chapter 8
   Glossary for Chapter 8

- Jules Berman

key words: computer science, data analysis, data repurposing, data simplification, data wrangling, information science, simplifying data, taming data, jules j berman

Thursday, March 3, 2016

Data Simplification: Contents

On March 23, my book, Data Simplification: Taming Information With Open Source Tools, will be published.

Here is a preview of the contents:

TABLE OF CONTENTS

Chapter 0. Preface
   References for Preface
   Glossary for Preface

Chapter 1. The Simple Life
   Section 1.1. Simplification drives scientific progress
   Section 1.2. The human mind is a simplifying machine
   Section 1.3. Simplification in Nature
   Section 1.4. The Complexity Barrier
   Section 1.5. Getting ready
   Open Source Tools for Chapter 1
      Perl
      Python
      Ruby
      Text Editors
      OpenOffice
      Command line utilities
      Cygwin, Linux emulation for Windows
      DOS batch scripts
      Linux bash scripts
      Interactive line interpreters
      Package installers
      System calls
References for Chapter 1
   Glossary for Chapter 1

Chapter 2. Structuring Text
   Section 2.1. The Meaninglessness of free text
   Section 2.2. Sorting text, the impossible dream
   Section 2.3. Sentence Parsing
   Section 2.4. Abbreviations
   Section 2.5. Annotation and the simple science of metadata
   Section 2.6. Specifications Good, Standards Bad
   Open Source Tools for Chapter 2
      ASCII
      Regular expressions
      Format commands
      Converting non-printable files to plain-text
      Dublin Core
   References for Chapter 2
   Glossary for Chapter 2

Chapter 3. Indexing Text
   Section 3.1. How Data Scientists Use Indexes
   Section 3.2. Concordances and Indexed Lists
   Section 3.3. Term Extraction and Simple Indexes
   Section 3.4. Autoencoding and Indexing with Nomenclatures
   Section 3.5. Computational Operations on Indexes
   Open Source Tools for Chapter 3
      Word lists
      Doublet lists
      Ngram lists
   References for Chapter 3
   Glossary for Chapter 3

Chapter 4. Understanding Your Data
   Section 4.1. Ranges and Outliers
   Section 4.2. Simple Statistical Descriptors
   Section 4.3. Retrieving Image Information
   Section 4.4. Data Profiling
   Section 4.5. Reducing data
   Open Source Tools for Chapter 4
      Gnuplot
      MatPlotLib
      R, for statistical programming
      Numpy
      Scipy
      ImageMagick
      Displaying equations in LaTex
      Normalized compression distance
      Pearson's correlation
      The ridiculously simple dot product
   References for Chapter 4 
   Glossary for Chapter 4

Chapter 5. Identifying and Deidentifying Data
   Section 5.1. Unique Identifiers
   Section 5.2. Poor Identifiers, Horrific Consequences
   Section 5.3. Deidentifiers and Reidentifiers
   Section 5.4. Data Scrubbing
   Section 5.5. Data Encryption and Authentication
   Section 5.6. Timestamps, Signatures, and Event Identifiers
   Open Source Tools for Chapter 5
      Pseudorandom number generators
      UUID
      Encryption and decryption with OpenSSL
      One-way hash implementations
      Steganography
   References for Chapter 5
   Glossary for Chapter 5

Chapter 6. Giving Meaning to Data
   Section 6.1. Meaning and Triples
   Section 6.2. Driving Down Complexity with Classifications
   Section 6.3. Driving Up Complexity with Ontologies
   Section 6.4. The unreasonable effectiveness of classifications
   Section 6.5. Properties that Cross Multiple Classes
   Open Source Tools for Chapter 6
      Syntax for triples
      RDF Schema
      RDF parsers
      Visualizing class relationships
   References for Chapter 6
   Glossary for Chapter 6

Chapter 7. Object-oriented data
   Section 7.1. The Importance of Self-explaining Data
   Section 7.2. Introspection and Reflection
   Section 7.3. Object-Oriented Data Objects
   Section 7.4. Working with Object-Oriented Data
   Open Source Tools for Chapter 7
      Persistent data
      SQLite databases
   References for Chapter 7
   Glossary for Chapter 7

Chapter 8. Problem simplification
   Section 8.1. Random numbers
   Section 8.2. Monte Carlo Simulations
   Section 8.3. Resampling and Permutating
   Section 8.4. Verification, Validation, and Reanalysis
   Section 8.5. Data Permanence and Data Immutability
   Open Source Tools for Chapter 8
      Burrows Wheeler transform
      Winnowing and chaffing
   References for Chapter 8
   Glossary for Chapter 8

Over the next few weeks, I will be blogging on topics selected from Data Simplification: Taming Information With Open Source Tools. I hope I can convince you that this is a book worth reading.

- Jules Berman

key words: computer science, data analysis, data repurposing, data simplification, data wrangling, information science, simplifying data, taming data, jules j berman

Blog readers can use the discount code: COMP315 for a 30% discount, at checkout.

Wednesday, March 2, 2016

Data Simplification: Why Bother?

"Order and simplification are the first steps toward the mastery of a subject." -Thomas Mann

Complex data is difficult to understand and analyze. Large data projects, using complex sets of data, are likely to fail; furthermore, the more money spent on a data project, the greater the likelihood of failure (1), (2), (3), (4), (5), (6), (7), (8), (9). What is true for data projects is also true in the experimental sciences; large and complex projects are often unrepeatable (10), (11), (12), (13), (14), (15), (16), (17), (18), (19), (20), (21), (22), (23), (24), (25), (26), (27), (28). Basically, complexity is something that humans have never mastered. As a species, we work much better when things are made simple.

Intelligent data scientists soon learn that it is nearly impossible to conduct competent and reproducible analyses of highly complex data. Inevitably, something always goes wrong. This book was written to provide a set of general principles and methods for data simplification. Here are the points that establish the conceptual framework of Data simplification: taming information with open source tools:

1) The first step in data analysis is data simplification. Most modern data scientists can expect that the majority of their time will be spent collecting, organizing, annotating, and simplifying their data, preparatory to analysis. A relatively small fraction of their time will be spend directly analyzing the data. When data has been simplified, successful analysis can proceed using standard computational methods, on standard computers.

2) Results obtained from unsimplified projects are nearly always irreproducible. Hence, the results of analyses on complex data cannot be validated. Conclusions that cannot be validated have no scientific value.

3) Data simplification is not simple. There is something self-defeating about the term, "data simplification". The term seems to imply a dumbing down process wherein naturally complex concepts are presented in a manner that is palatable to marginally competent scientists. Nothing can be further from the truth. Creating overly complex data has always been the default option for lazy-minded or cavalier scientists who lacked the will or the talent to produce a simple, well-organized, and well-annotated collection of data. The act of data simplification will always be one of the most challenging tasks facing data scientists, often requiring talents drawn from multiple disciplines. The sad truth is that there are very few data professionals who are competent to perform data simplification; and fewer still educators who can adequately teach the subject.

4) No single software application will solve your data simplification needs (29). Applications that claim to do everything for the user are, in most instances, applications that require the user to do everything for the application. The most useful software solutions often come in the form of open source utilities, designed to perform one method, very well, and very fast. In this book, dozens of freely available utilities are demonstrated.

5) Data simplification tools are data discovery tools, in the hands of creative individuals. The act of data simplification always gives the scientist a better understanding of the meaning of the data. Data that has been organized and annotated competently provides us with new questions, new hypotheses, and new approaches to problem solving. Data that is complex only provides headaches.

6) Data simplification is a prerequisite for data preservation. Data that has not been simplified has no useful shelf-life. After a data project has ended, nobody will be able to understand what was done. This means no future projects will build upon the original data, or find new purposes for the data. Moreover, conclusions drawn from the original data will never be verified or validated. This means that when you do not simplify your data, your colleagues will not accept your conclusions. Those who understand the principles and practice of data simplification will produce credible data that can be validated and repurposed.

7) Data simplification saves money. Data simplification often involves developing general solutions that apply to classes of data. By eliminating the cost of using made-to-order proprietary software, data scientists can increase their productivity and reduce their expenses.

8) Learning the methods and tools of data simplification is a great career move. Data simplification is the the next big thing in the data sciences. The most thoughtful employers understand that it's not always about keeping it simple. More often, it's about making it simple.

9) Data scientists should have familiarity with more than one programming language. Although one high-level language has much the same functionality as another, each language may have particular advantages in different situations. For example, a programmer may prefer Perl when her tasks involve text parsing and string pattern matches. Another programmer might prefer Python if she requires a variety of numeric or analytic functions and a smooth interface to a graphing tool. Programmers who work with classes of data objects, or who need to model new classifications, might prefer the elegant syntax and rich class libraries available in Ruby. Books that draw on a single programming language run the risk of limiting the problem-solving options of their readers. Although there are many high-quality programming languages, I have chosen Perl, Python and Ruby as the demonstration languages for this book. Each of these popular languages is free, open source, and can be installed easily and quickly on virtually any operating system. By offering solutions in several different programming languages, this book may serve as a sort of Rosetta stone for data scientists who must work with data structures produced in different programming environments.

Blog readers can use the discount code: COMP315 for a 30% discount, at checkout.

References:

[1] Kappelman LA, McKeeman R, Lixuan Zhang L. Early warning signs of IT project failure: the dominant dozen. Information Systems Management 23:31-36, 2006.

[2] Arquilla J. The Pentagon's biggest boondoggles. The New York Times (Opinion Pages) March 12, 2011.

[3] Lohr S. Lessons From Britain's Health Information Technology Fiasco. The New York Times Sept. 27, 2011.

[4] Dismantling the NHS national programme for IT. Department of Health Media Centre Press Release. September 22, 2011. Available from: http://mediacentre.dh.gov.uk/2011/09/22/dismantling-the-nhs-national-programme-for-it/ viewed June 12, 2012.

[5] Whittaker Z. UK's delayed national health IT programme officially scrapped. ZDNet September 22, 2011.

[6] Lohr S. Google to end health records service after it fails to attract users. The New York Times Jun 24, 2011.

[7] Schwartz E. Shopping for health software, some doctors get buyer's remorse. The Huffington Post Investigative Fund Jan 29, 2010.

[8] Heeks R, Mundy D, Salazar A. Why health care information systems succeed or fail. Institute for Development Policy and Management, University of Manchester, June 1999 Available from: http://www.sed.manchester.ac.uk/idpm/research/publications/wp/igovernment/igov_wp09.htm, viewed July 12, 2012.

[9] Beizer B. Software Testing Techniques. Van Nostrand Reinhold; Hoboken, NJ 2 edition, 1990.

[10] Unreliable research: Trouble at the lab. The Economist October 19, 2013.

[11] Kolata G. Cancer fight: unclear tests for new drug. The New York Times April 19, 2010.

[12] Ioannidis JP. Why most published research findings are false. PLoS Med 2:e124, 2005.

[13] Baker M. Reproducibility crisis: Blame it on the antibodies. Nature 521:274-276, 2015.

[14] Naik G. Scientists' Elusive Goal: Reproducing Study Results. Wall Street Journal December 2, 2011.

[15] Innovation or Stagnation: Challenge and Opportunity on the Critical Path to New Medical Products. U.S. Department of Health and Human Services, Food and Drug Administration, 2004.

[16] Hurley D. Why Are So Few Blockbuster Drugs Invented Today? The New York Times November 13, 2014.

[17] Angell M. The Truth About the Drug Companies. The New York Review of Books Vol 51, July 15, 2004.

[18] Crossing the Quality Chasm: A New Health System for the 21st Century. Quality of Health Care in America Committee, editors. Institute of Medicine, Washington, DC., 2001.

[19] Wurtman RJ, Bettiker RL. The slowing of treatment discovery, 1965-1995. Nat Med 2:5-6, 1996.

[20] Ioannidis JP. Microarrays and molecular research: noise discovery? The Lancet 365:454-455, 2005.

[21] Weigelt B, Reis-Filho JS. Molecular profiling currently offers no more than tumour morphology and basic immunohistochemistry. Breast Cancer Research 12:S5, 2010.

[22] Personalised medicines: hopes and realities. The Royal Society, London, 2005.Available from: https://royalsociety.org/~/media/Royal_Society_Content/policy/publications/2005/9631.pdf, viewed Jan 1, 2015.

[23] Vlasic B. Toyota's slow awakening to a deadly problem. The New York Times, February 1, 2010.

[24] Lanier J. The complexity ceiling. In: Brockman J, ed. The next fifty years: science in the first half of the twenty-first century. Vintage, New York, pp 216-229, 2002.

[25] Ecker JR, Bickmore WA, Barroso I, Pritchard JK, Gilad Y, Segal E. Genomics: ENCODE explained. Nature 489:52-55, 2012.

[26] Rosen JM, Jordan CT. The increasing complexity of the cancer stem cell paradigm. Science 324:1670-1673, 2009.

[27] Labos C. It Ain't Necessarily So: Why Much of the Medical Literature Is Wrong. Medscape News and Perspectives. September 09, 2014

[28] Gilbert E, Strohminger N. We found only one-third of published psychology research is reliable - now what? The Conversation. August 27, 2015. Available at: http://theconversation.com/we-found-only-one-third-of-published-psychology-research-is-reliable-now-what-46596, viewed on August 27,2015.

[29] Brooks FP. No silver bullet: essence and accidents of software engineering. Computer 20:10-19, 1987. Comment. This early (1987) paper tackles the problem of software complexity, emphasizing the importance of software design. He also suggests that one of the best ways for institutions to achieve computer productivity is to provide staff with basic applications (e.g., wordprocessor, spreadsheet, statistical packages) and train them with simple programming skills.