Those of you who are computer-oriented know that data analysis typically takes much less time and effort than data preparation. Moreover, if you make a mistake in your data analysis, you can often just repeat the process, using different tools, or a fresh approach to your original question. As long as the data is prepared properly, you and your colleagues can re-analyze your data to your heart's content. Contrariwise, if your data is not prepared in a manner that supports sensible analysis, there's little you can do to extricate yourself from the situation. For this reason, data preparation is, in my experience, much more important than data analysis.
Throughout my career, I've relied on simple open source utilities and short scripts to simplify my data, producing products that were self-explanatory, permanent, and that could be merged with other types of data. Hence, my book.
Data Simplification: Taming Information With Open Source Tools
Publisher: Morgan Kaufmann; 1 edition (March 23, 2016)
Paperback: 398 pages
Dimensions: 7.5 x 9.2 inches
Chapter 1, The Simple Life, explores the thesis that complexity is the rate-limiting factor in human development. The greatest advances in human civilization and the most dramatic evolutionary improvements in all living organisms have followed the acquisition of methods that reduce or eliminate complexity.
Chapter 2, Structuring Text, reminds us that most of the data on the Web today is unstructured text, produced by individuals, trying their best to communicate with one another. Data simplification often begins with textual data. This chapter provides readers with tools and strategies for imposing some basic structure on free-text.
Chapter 3, Indexing Text, describes the often undervalued benefits of indexes. An index, aided by proper annotation of data, permits us to understand data in ways that were not anticipated when the original content was collected. With the use of computers, multiple indexes designed for differentpurposes, can be created for a single document or data set. As data accrues, indexes can be updated. When data sets are combined, their respective indexes can be merged. A good way of thinking about indexes is that the document contains all of the complexity; the index contains all of the simplicity. Data scientists who understand how to create and use indexes will be in the best position to search, retrieve, and analyze textual data. Methods are provided for automatically creating customized indexes designed for specific analytic pursuits and for binding index terms to standard nomenclatures.
Chapter 4, Understanding Your Data, describes how data can be quickly assessed, prior to formal quantitative analysis, to develop some insight into what the data means. A few simple visualization tricks and simple statistical descriptors can greatly enhance a data scientist’s understanding of complex and large data sets. Various types of data objects, such as text files, images, and time-series data, can be profiled with a summary signature that captures the key features that contribute to the behavior and content of the data object. Such profiles can be used to find relationships among different data objects, or to determine when data objects are not closely related to one another.
Chapter 5, Identifying and Deidentifying Data, tackles one of the most under-appreciated and least understood issues in data science. Measurements, annotations, properties, and classes of information have no informational meaning unless they are attached to an identifier that distinguishes one data object from all other data objects, and that links together all of the information that has been or will be associated with the identified data object. The method of identification and the selection of objects and classes to be identified relates fundamentally to the organizational model of complex data. If the simplifying step of data identification is ignored or implemented improperly, data cannot be shared, and conclusions drawn from the data cannot be believed. All well-designed information systems are, at their heart, identification systems: ways of naming data objects so that they can be retrieved. Only well-identified data can be usefully deidentified. This chapter discusses methods for identifying data and deidentifying data.
Chapter 6, Giving Meaning to Data, explores the meaning of meaning, as it applies to computer science. We shall learn that data, by itself, has no meaning. It is the job of the data scientist to assign meaning to data, and this is done with data objects, triples, and classifications (see Glossary items, Data object, Triple, Classification, Ontology). Unfortunately, coursework in the information sciences often omits discussion of the critical issue of "data meaning"; advancing from data collection to data analysis without stopping to design data objects whose relationships to other data objects are defined and discoverable. In this chapter, readers will learn how to prepare and classify meaningful data.
Chapter 7, Object-Oriented Data, shows how we can understand data, using a few elegant computational principles. Modern programming languages, particularly object-oriented programming languages, use introspective data (ie, the data with which data objects describe themselves) to modify the execution of a program at run-time; an elegant process known as reflection. Using introspection and reflection, programs can integrate data objects with related data objects. The implementations of introspection, reflection and integration, are among the most important achievements in the field of computer science.
Chapter 8, Problem Simplification, demonstrates that it is just as important to simplify problems as it is to simplify data. This final chapter provides simple but powerful methods for analyzing data, without resorting to advanced mathematical techniques. The use of random number generators to simulate the behavior of systems, and the application of Monte Carlo, resampling, and permutative methods to a wide variety of common problems in data analysis, will be discussed. The importance of data reanalysis, following preliminary analysis, is emphasized.
TABLE OF CONTENTS Chapter 0. Preface References for Preface Glossary for Preface Chapter 1. The Simple Life Section 1.1. Simplification drives scientific progress Section 1.2. The human mind is a simplifying machine Section 1.3. Simplification in Nature Section 1.4. The Complexity Barrier Section 1.5. Getting ready Open Source Tools for Chapter 1 Perl Python Ruby Text Editors OpenOffice Command line utilities Cygwin, Linux emulation for Windows DOS batch scripts Linux bash scripts Interactive line interpreters Package installers System calls References for Chapter 1 Glossary for Chapter 1 Chapter 2. Structuring Text Section 2.1. The Meaninglessness of free text Section 2.2. Sorting text, the impossible dream Section 2.3. Sentence Parsing Section 2.4. Abbreviations Section 2.5. Annotation and the simple science of metadata Section 2.6. Specifications Good, Standards Bad Open Source Tools for Chapter 2 ASCII Regular expressions Format commands Converting non-printable files to plain-text Dublin Core References for Chapter 2 Glossary for Chapter 2 Chapter 3. Indexing Text Section 3.1. How Data Scientists Use Indexes Section 3.2. Concordances and Indexed Lists Section 3.3. Term Extraction and Simple Indexes Section 3.4. Autoencoding and Indexing with Nomenclatures Section 3.5. Computational Operations on Indexes Open Source Tools for Chapter 3 Word lists Doublet lists Ngram lists References for Chapter 3 Glossary for Chapter 3 Chapter 4. Understanding Your Data Section 4.1. Ranges and Outliers Section 4.2. Simple Statistical Descriptors Section 4.3. Retrieving Image Information Section 4.4. Data Profiling Section 4.5. Reducing data Open Source Tools for Chapter 4 Gnuplot MatPlotLib R, for statistical programming Numpy Scipy ImageMagick Displaying equations in LaTex Normalized compression distance Pearson's correlation The ridiculously simple dot product References for Chapter 4 Glossary for Chapter 4 Chapter 5. Identifying and Deidentifying Data Section 5.1. Unique Identifiers Section 5.2. Poor Identifiers, Horrific Consequences Section 5.3. Deidentifiers and Reidentifiers Section 5.4. Data Scrubbing Section 5.5. Data Encryption and Authentication Section 5.6. Timestamps, Signatures, and Event Identifiers Open Source Tools for Chapter 5 Pseudorandom number generators UUID Encryption and decryption with OpenSSL One-way hash implementations Steganography References for Chapter 5 Glossary for Chapter 5 Chapter 6. Giving Meaning to Data Section 6.1. Meaning and Triples Section 6.2. Driving Down Complexity with Classifications Section 6.3. Driving Up Complexity with Ontologies Section 6.4. The unreasonable effectiveness of classifications Section 6.5. Properties that Cross Multiple Classes Open Source Tools for Chapter 6 Syntax for triples RDF Schema RDF parsers Visualizing class relationships References for Chapter 6 Glossary for Chapter 6 Chapter 7. Object-oriented data Section 7.1. The Importance of Self-explaining Data Section 7.2. Introspection and Reflection Section 7.3. Object-Oriented Data Objects Section 7.4. Working with Object-Oriented Data Open Source Tools for Chapter 7 Persistent data SQLite databases References for Chapter 7 Glossary for Chapter 7 Chapter 8. Problem simplification Section 8.1. Random numbers Section 8.2. Monte Carlo Simulations Section 8.3. Resampling and Permutating Section 8.4. Verification, Validation, and Reanalysis Section 8.5. Data Permanence and Data Immutability Open Source Tools for Chapter 8 Burrows Wheeler transform Winnowing and chaffing References for Chapter 8 Glossary for Chapter 8- Jules Berman
key words: computer science, data analysis, data repurposing, data simplification, data wrangling, information science, simplifying data, taming data, jules j berman