Paradoxes of Classification (and terrible Class definitions)

The formal systems that assign data objects to classes, and that relate classes to other classes, are known as ontologies. When the data within a Big Data resource is classified within an ontology, data analysts can determine whether observations on a single object will apply to other objects in the same class. Similarly, data analysts can begin to ask whether observations that hold true for a class of objects will relate to other classes of objects. Basically, ontologies help scientists fulfill one of their most important tasks; determining how things relate to other things.

A classification is a very simple form of ontology, in which each class is allowed to have only one parent class. To build a classification, the ontologist must do the following: 1) define classes (i.e., find the properties that define a class and extend to the subclasses of the class); 2) assign instances to classes; 3) position classes within the hierarchy; and 4) test and validate all the above.

The constructed classification becomes a hierarchy of data objects conforming to a set of principles:

• The classes (groups with members) of the hierarchy have a set of properties or rules that extend to every member of the class and to all of the subclasses of the class, to the exclusion of unrelated classes . A subclass is itself a type of class wherein the members have the defining class properties of the parent class plus some additional property(ies) specific for the subclass.

• In a hierarchical classification, each subclass may have no more than one parent class. The root (top) class has no parent class. The biological classification of living organisms is a hierarchical classification.
• At the bottom of the hierarchy is the class instance. For example, your copy of this book is an instance of the class of objects known as "books".
• Every instance belongs to exactly one class.
• Instances and classes do not change their positions in the classification. As examples, a horse never transforms into a sheep, and a book never transforms into a harpsichord.
• The members of classes may be highly similar to one another, but their similarities result from their membership in the same class (i.e., conforming to class properties), and not the other way around (i.e., similarity alone cannot define class inclusion).

Classifications are always simple; the parental classes of any instance of the classification can be traced as a simple, non-branched list, ascending through the class hierarchy. As an example, here is the lineage for the domestic horse (Equus caballus), from the classification of living organisms:

Equus caballus
Equus subg. Equus
Equus
Equidae
Perissodactyla
Laurasiatheria
Eutheria
Theria
Mammalia
Amniota
Tetrapoda
Sarcopterygii
Euteleostomi
Teleostomi
Gnathostomata
Vertebrata
Craniata
Chordata
Deuterostomia
Coelomata
Bilateria
Eumetazoa
Metazoa
Fungi/Metazoa group
Eukaryota
cellular organisms

Taxonomists who view this lineage instantly grasp the place of domestic horses in the classification of all living organisms.

The rules for constructing classifications seem obvious and simplistic. Surprisingly, the task of building a logical, and self-consistent classification is extremely difficult. Most classifications are rife with logical inconsistencies and paradoxes. Let's look at a few examples.

In 1975, while touring the Bethesda, Maryland campus of the National Institutes of Health, I was informed that their Building 10, was the largest all-brick building in the world, providing a home to over 7 million bricks . Soon thereafter, an ambitious construction project was undertaken to greatly expand the size of Building 10. When the work was finished, building 10 was no longer the largest all-brick building in the world. What happened? The builders used material other than brick, and Building 10 lost its classification as an all-brick building, violating the immutability rule of class assignments.

Apparent paradoxes that plague any formal conceptualization of classifications are not difficult to find. Let's look at a few more examples.

Consider the geometric class of ellipses; planar objects in which the sum of the distances to two focal points is constant. Class Circle is a child of Class Ellipse, for which the two focal points of instance members occupy the same position, in the center, producing a radius of constant size. Imagine that Class Ellipse is provided with a class method called "stretch", in which the foci are moved further apart, thus producing flatter objects. When the parent class "stretch" method is applied to members of the Class Circle, the circle stops being a circle and becomes an ordinary ellipse. Hence the inherited "stretch" method forces members of Class Circle to transition out of their assigned class, violating the intransitive rule of classifications.

Let's look at the "Bag" class of objects. A "Bag" is a collection of objects, and the Class Bag is included in most object-oriented programming languages. A "Set" is also a collection of objects (i.e., a subclass of Bag), with the special feature that duplicate instances are not permitted. For example, if Kansas is a member of the set of U.S. States, then you cannot add a second state named "Kansas" to the set. If Class Bag were to have an "increment" method, that added "1" to the total count of objects in the bag, whenever an object is added to Class Bag, then the "increment" method would be inherited by all of the subclasses of Class Bag, including Class Set. But Class Set cannot increase in size when duplicate items are added. Hence, inheritance creates a paradox in the Class Set.

How does a data scientist deal with class objects that disappear from their assigned class and reappear elsewhere? In the examples discussed here, we saw the following:

1. Building 10 at NIH was defined as the largest all-brick building in the world. Strictly speaking, Building 10 was a structure, and it had a certain weight and dimensions, and it was constructed of brick. "Brick" is an attribute or property of buildings, and properties cannot form the basis of a class of building, if they are not a constant feature shared by all members of the class (i.e., some buildings have bricks; others do not). Had we not conceptualized an "all-brick" class of building, we would have avoided any confusion.

2. Class Circle qualified as a member of Class Ellipse, because a circle can be imagined as an ellipse whose two focal points happen to occupy the same location. Had we defined Class Ellipse to specify that class members must have two separate focal points, we could have excluded circles from class Ellipse. Hence, we could have safely included the stretch method in Class Ellipse without creating a paradox.

3. Class Set was made a subset of Class Bag, but the increment method of class Bag could not apply to Class Set. We created Class Set without taking into account the basic properties of Class Bag, which must apply to all its subclasses. Perhaps it would have been better if Class Set and Class Bag were created as children of Class Collection; each with its own set of properties.

Worst Class Definition Ever

The worst definition of a Class may have been that given to the Kingdom of Protozoa, defined as the class of one-celled eukaryotic organisms. The problem here is that all of the classes of multicelled organisms (e.g., animals, plants and fungi) descended from classes of one-celled organisms. This means that Class Protozoa (defined as one-cell organisms) must exclude from its lineage all descendant classes that are multicellular. Hence, Kingdom Protozoa was given a definition that, paradoxically, excluded its own descendants. What there they thinking, back in the mid-19th century when Class Protozoa was conceived?

- Jules Berman

key words: classification, ontology, taxonomy, paradoxes, precision medicine, jules j berman Ph.D., M.D.

The Importance of Biological Taxonomy

Biological taxonomy is the scientific field dealing with the classification of living organisms. Non-biologists who give any thought to taxonomy, may think that the field is the dullest of the sciences. To the uninitiated, there is little difference between the life of a taxonomist and the life of a stamp collector. Nothing could be further from the truth. Taxonomy has become the grand unifying theory of the biological sciences. Efforts to sequence the genomes of prokaryotic, eukaryotic and viral species, thereby comparing the genomes of different classes of organisms, have revitalized the field of evolutionary taxonomy (phylogenetics). The analysis of normal and abnormal homologous genes in related classes of organisms have inspired new disease treatments targeted against specific molecules and pathways characteristic of species or classes or organisms. Students who do not understand the principles of modern taxonomy have little chance of perceiving the connections between medicine, genetics, pharmacology, or pathology, to say nothing of clinical microbiology.

Here are two of the specific advantages of learning the taxonomy of infectious diseases.

1. As a method to drive down the complexity of medical microbiology

Learning all the infectious diseases of humans is an impossible task. As the number of chronically ill and immune-compromised patients has increased, so have the number of opportunistic pathogens. As global transportation has become commonplace, the number of exotic infections spread worldwide has also increased (See Glossary item, Exotic diseases in the United States). A few decades ago, infectious disease experts were expected to learn a few hundred infectious diseases. Today, there are over 1400 organisms that can cause diseases in humans, and the number is climbing rapidly, while the techniques to diagnose and treat these organisms are constantly improving. Textbooks cannot cover all these organisms in sufficient detail to provide healthcare workers with the expertise to provide adequate care to their patients.

How can any clinician learn all that is needed to provide competent care to patients? The first step in understanding infectious diseases is to understand the classification of pathogenic organisms. Every known disease-causing organisms has been assigned to one of 40 well-defined classes of organisms, and each class fits within a simple ancestral lineage. This means that every known pathogenic organism inherits certain properties from its ancestral classes and shares these properties with the other members of its own class. When you learn the class properties, along with some basic information about the infectious members of the classes, you gain a comprehensive understanding of medical microbiology.

2. As protection against professional obsolescence

There seems to be so much occurring the the biological sciences, it is just impossible to keep on top of things. With each passing day, you feel less in tune with modern science, and you wish you could return to a time when a few fundamental principles grounded your chosen discipline. You will be happy to learn that science is all about finding generalizations among data or among connected systems (i.e., reducing the complexity of data or finding simple explanations for systems of irreducible complexity). Much, if not all, of the perceived complexity of the biological sciences derives from the growing connections of once-separate disciplines: cell biology, ecology, evolution, climatology, molecular biology, pharmacology, genetics, computer sciences, paleontology, pathology, statistics, and so on. Scientists today must understand many different fields, and they must be willing and able to absorb additional disciplines, throughout their careers. As each field of science becomes entangled with other the seemingly arcane field of biological taxonomy has gained prominence because it occupies the intellectual core of virtually every biological field.

Modern biology is data-driven. A deluge of organism-based genomic, proteomic, metabolomic and other "omic" data is flooding our data banks and drowning our scientists. This data will have limited scientific value if we cannot find a way to generalize the data collected for each organism to the data collected in other organisms. Taxonomy is the scientific method that reveals how different organisms are related. Without taxonomy, data has no biological meaning.

The discoveries that scientists make in the future will come from questions that arise during the construction and refinement of biological taxonomy. In the case of infectious diseases, when we find a trait that informs us that what we thought was a single species is actually two species, it permits us to develop treatments optimized for each species, and to develop new methods to monitor and control the spread of both organisms. When we correctly group organisms within a common class, we can test and develop new drugs that are effective against all of the organisms within the class, particularly if those organisms are characterized by a molecule, pathway or trait that is specifically targeted by a drug. Terms used in diverse sciences, such as homology, metabolic pathway, target molecule, acquired resistance, developmental stage, cladistics, monophyly, model organism, class property, phylogeny, all derive their meaning and their utility from biological taxonomy. When you grasp the general organization of living organisms, you will understand how different scientific fields relate to each other, thus avoiding professional obsolescence.

- Jules Berman (copyrighted material)

key words: taxonomy, evolution, classification, data organization, jules j berman

Unclassifiable objects

Classifications create a class for every object and taxonomies assign each and every object to its correct class. This means that a classification is not permitted to contain unclassified objects; a condition that puts fussy taxonomists in an untenable position. Suppose you have an object, and you simply do not know enough about the object to confidently assign it to a class. Or, suppose you have an object that seems to fit more than one class, and you can't decide which class is the correct class. What do you do? Historically, scientists have resorted to creating a "miscellaneous" class into which otherwise unclassifiable objects are given a temporary home, until more suitable accommodations can be provided. I have spoken with numerous data managers, and everyone seems to be of a mind that "miscellaneous" classes, created as a stopgap measure, serve a useful purpose. Not so. Historically, the promiscuous application of "miscellaneous" classes have proven to be a huge impediment to the advancement of science. In the case of the classification of living organisms, the class of protozoans stands as a case in point. Ernst Haeckel, a leading biological taxonomist in his time, created the Kingdom Protista (i.e., protozoans), in 1866, to accommodate a wide variety of of simple organisms with superficial commonalities. Haeckel himself understood that the protists were a blended class that included unrelated organisms, but he believed that further study would resolve the confusion. In a sense, he was right, but the process took much longer than he had anticipated; occupying generations of taxonomists over the following 150 years. Today, Kingdom Protista no longer exists. Its members have been reassigned to various classes of unicellular eukaryotes. Nonetheless, textbooks of microbiology still describe the protozoans, just as though this name continued to occupy a legitimate place among terrestrial organisms. In the meantime, therapeutic opportunities for eradicating so-called protozoal infections, using class-targeted agents, have no doubt been missed (1). You might think that the creation of a class of living organisms, with no established scientific relation to the real world, was a rare and ancient event in the annals of biology, having little or no chance of being repeated. Not so. A special pseudoclass of fungi, deuteromyctetes (spelled with a lowercase "d", signifying its questionable validity as a true biologic class) has been created to hold fungi of indeterminate speciation. At present, there are several thousand such fungi, sitting in a taxonomic limbo, waiting to be placed into a definitive taxonomic class (2), (1).

[1] Berman JJ. Taxonomic Guide to Infectious Diseases: Understanding the Biologic Classes of Pathogenic Organisms. Academic Press, Waltham, 2012.

[2] Guarro J, Gene J, Stchigel AM. Developments in fungal taxonomy. Clinical Microbiology Reviews 12:454-500, 1999.

- Jules Berman (copyrighted material)

key words: classifications, ontology, classes, taxonomy, jules j berman

CLASSIFICATIONS, THE SIMPLEST OF ONTOLOGIES

Here is a short excerpt from my book Principles of Big Data: Preparing, Sharing, and Analyzing Complex Information

---

CLASSIFICATIONS, THE SIMPLEST OF ONTOLOGIES

The human brain is constantly processing visual and other sensory information collected from the environment. When we walk down the street, we see images of concrete and asphalt and millions of blades of grass, and birds, and dogs, and other persons, and so on. Every step we take conveys a new world of sensory input. How can we process it all? The mathematician and philosopher Karl Pearson has (1857 - 1936) has likened the human mind to a "sorting machine". We take a stream of sensory information and sort it into objects; we then we collect the individual objects into general classes. The green stuff on the ground is classified as "grass," and the grass is subclassified under some larger grouping, such as "plants." A flat stretch of asphalt and concrete may be classified as a "road" and the road might be subclassified under "man-made constructions". If we lacked a culturally-determined classification of objects for our world, we would be overwhelmed by sensory input, and we would have no way to remember what we see, and no way to draw general inferences about anything. Simply put, without our ability to classify, we would not be human.

Every culture has some particular way to impose a uniform way of perceiving the environment. In English-speaking cultures, the term "hat" denotes a universally recognized object. Hats may be composed of many different types of materials, and they may vary greatly in size, weight, and shape. Nonetheless, we can almost always identify a hat when we see one, and we can distinguish a hat from all other types of objects. An object is not classified as a hat simply because it shares a few structural similarities with other hats. A hat is classified as a hat because it has a class relationship; all hats are items of clothing that fit over the head. Likewise, all biological classifications are built by relationships, not by similarities.

Aristotle was one of the first experts in classification. His greatest insight came when he correctly identified a dolphin as a mammal. Through observation, he knew that a large group of animals was distinguished by a gestational period in which a developing embryo is nourished by a placenta, and the offspring are delivered into the world as formed, but small versions of the adult animals (i.e., not as eggs or larvae), and the newborn animals feed from milk excreted from nipples, overlying specialized glandular organs (mammae). Aristotle knew that these features, characteristic of mammals, were absent in all other types of animals. He also knew that dolphins had all these features; fish did not. He correctly reasoned that dolphins were a type of mammal, not a type of fish. Aristotle was ridiculed by his contemporaries for whom it was obvious that dolphins were a type of fish. Unlike Aristotle, they based their classification on similarities, not on relationships. They saw that dolphins looked like fish and dolphins swam in the ocean like fish, and this was all the proof they needed to conclude that dolphins were indeed fish. For about two thousand years following the death of Aristotle, biologists persisted in their belief that dolphins were a type of fish. For the past several hundred years, biologists have acknowledged that Aristotle was correct after all; dolphins are mammals. Aristotle discovered and taught the most important principle of classification; that classes are built on relationships among class members; not by counting similarities. We will see in later chapters, that methods of grouping data objects by similarity can be very misleading, and should not be used as the basis for constructing a classification or an ontology.

A classification is a very simple form of ontology, in which each class is limited to one parent class. To build a classification, the ontologist must do the following: 1) define classes (i.e., find the properties that define a class and extend to the subclasses of the class); 2) assign instances to classes; 3) position classes within the hierarchy; and 4) test and validate all the above.

The constructed classification becomes a hierarchy of data objects conforming to a set of principles:

1. The classes (groups with members) of the hierarchy have a set of properties or rules that extend to every member of the class and to all of the subclasses of the class, to the exclusion of unrelated classes . A subclass is itself a type of class wherein the members have the defining class properties of the parent class plus some additional property(ies) specific for the subclass.

2. In a hierarchical classification, each subclass may have no more than one parent class. The root (top) class has no parent class. The biological classification of living organisms is a hierarchical classification.

3. At the bottom of the hierarchy is the class instance. For example, your copy of this book is an instance of the class of objects known as "books".

4. Every instance belongs to exactly one class.

5. Instances and classes do not change their positions in the classification. As examples, a horse never transforms into a sheep, and a book never transforms into a harpsichord.

6. The members of classes may be highly similar to one another, but their similarities result from their membership in the same class (i.e., conforming to class properties), and not the other way around (i.e., similarity alone cannot define class inclusion).

Classifications are always simple; the parental classes of any instance of the classification can be traced as a simple, non-branched list, ascending through the class hierarchy. As an example, here is the lineage for the domestic horse (Equus caballus), from the classification of living organisms:

Equus caballus
Equus subg. Equus
Equus
Equidae
Perissodactyla
Laurasiatheria
Eutheria
Theria
Mammalia
Amniota
Tetrapoda
Sarcopterygii
Euteleostomi
Teleostomi
Gnathostomata
Vertebrata
Craniata
Chordata
Deuterostomia
Coelomata
Bilateria
Eumetazoa
Metazoa
Fungi/Metazoa group
Eukaryota
cellular organisms

The words in this zoologic lineage may seem strange to laypersons, but taxonomists who view this lineage instantly grasp the place of domestic horses in the classification of all living organisms.

A classification is a list of every member class, along with their relationships to other classes. Because each class can have only one parent class, a complete classification can be provided when we list all the classes, adding the name of the parent class for each class on the list. For example, a few lines of the classification of living organisms might be:

Craniata, subclass of Chordata
Chordata, subclass of Duterostomia
Deuterostomia, subclass of Coelomata
Coelomata, subclass of Bilateria
Bilateria, sublcass of Eumetazoa

Given the name of any class, a programmer can compute (with a few lines of code), the complete ancestral lineage for the class, by iteratively finding the parent class assigned to each ascending class.

A taxonomy is a classification with the instances "filled in." This means that for each class in a taxonomy, all the known instances (i.e., member objects) are explicitly listed. For the taxonomy of living organisms, the instances are named species. Currently, there are several million named species of living organisms, and each of these several million species is listed under the name of some class included in the full classification.

Classifications drive down the complexity of their data domain, because every instance in the domain is assigned to a single class, and every class is related to the other classes through a simple hierarchy.

It is important to distinguish a classification system from an identification system. An identification system puts a data object into its correct slot within the classification. For example, a fingerprint matching system may look for a set of features that puts a fingerprint into a special subclass of all fingerprint, but the primary goal of fingerprint matching is to establish the identity of an instance (i.e., to show that two sets of fingerprints belong to the same person). In the realm of medicine, when a doctor renders a diagnosis on a patient's diseases, she is not classifying the disease; she is finding the correct slot within the pre-existing classification of diseases that holds her patient's diagnosis.

-Jules Berman (Copyrighted material)

key words: classification, ontology, taxonomy, jules j berman

Perl script for extracting lineages of organisms in EBI Taxonomy

In the past two blogs, I presented Ruby and Python scripts to create phylogentic lineages for species included in taxonomy.dat. Here is the equivalent project, in Perl.

Taxonomy.dat is a large, publicly available list of organisms. The file is available from the European Bioinformatics Institute (EBI). It contains over 400,000 species:

[A sample record in Taxonomy.dat]
ID : 438
PARENT ID : 434
RANK : species
GC ID : 11
SCIENTIFIC NAME : Acetobacter pasteurianus
SYNONYM : Acetobacter lovaniense
SYNONYM : Acetobacter alcoholophilus
SYNONYM : Acetobacter pasteurianus (Hansen 1879) Beijerinck and Folpmers 1916
SYNONYM : "Ulvina pasteuriana" (Hansen 1879) Pribram 1933
SYNONYM : "Pseudomonas pomi" Cole 1959
SYNONYM : "Mycoderma pasteurianum" Hansen 1879
SYNONYM : Acetobacter pasteurianus ascendens
SYNONYM : Acetobacter pasteurianus paradoxus
SYNONYM : "Acetobacter alcoholophilus" Kozulis and Parsons 1958
SYNONYM : "Acetobacter kutzigianus" (sic) (Hansen 1894) Bergey et al. 1923
SYNONYM : "Acetobacter mobile" (sic) Tosic and Walker 1944
SYNONYM : "Acetobacter vini-aceti" (Henneberg 1906) Shimwell 1948
SYNONYM : "Bacterium vini-aceti" Henneberg 1906
SYNONYM : "Bacterium rancens" Beijerinck 1898
SYNONYM : "Bacillus kuttingianum" (sic) (Hansen 1894) Takahashi 1906
SYNONYM : "Bacillus pasteurianus" (Hansen 1879) Flugge 1886
SYNONYM : "Bacterium pastorianum" (Hansen 1879) Zopf 1883
SYNONYM : "Bacterium kutzingianum" Hansen 1894
SYNONYM : "Bacteriopsis pasteuriana" (Hansen 1879) Trevisan 1885
SYNONYM : Acetobacter agglutinans
SYNONYM : Acetobacter acidum-mucosum
SYNONYM : "Bacillus pasteurianus" (Hansen 1879) Fl gge 1886
SYNONYM : "Acetobacter turbidans" Cosbie et al. 1942
SYNONYM : Acetobacter kutzigianus
SYNONYM : Acetobacter mobile
SYNONYM : Acetobacter turbidans
SYNONYM : Acetobacter vini-aceti
SYNONYM : Acetobacter pasteurianus subsp. orleanensis
SYNONYM : Acetobacter pasteurianus orleanensis
SYNONYM : Bacillus kuttingianum
SYNONYM : "Acetobacter acidum-mucosum" (sic) Tosic and Walker 1950
SYNONYM : Bacteriopsis pasteuriana
SYNONYM : Bacterium kutzingianum
SYNONYM : Acetobacter rancens
SYNONYM : "Acetobacter agglutinans" Frateur 1950
SYNONYM : Ulvina pasteuriana
SYNONYM : Pseudomonas pomi
SYNONYM : Mycoderma pasteurianum
SYNONYM : Bacterium vini-aceti
SYNONYM : Bacterium pastorianum
SYNONYM : Bacterium rancens
INCLUDES : Acetobacter turbidans ATCC 9325
INCLUDES : Acetobacter turbidans ATCC9325
IN-PART : Bacillus pasteurianus
//

The taxonomy.dat file exceeds 100 megabytes in length.

The taxonomy.dat file is available for public download through anonymous ftp.

[ftp://ftp.ebi.ac.uk/pub/databases/taxonomy/]

Information about the taxonomy.dat file is found at:

[http://www.ebi.ac.uk/msd-srv/docs/dbdoc/ref_taxonomy.html]
Notice that the sample entry (above) provides an ID number for the entry organism, and for it's parent class. Since every organism and class has a parent, you can write a script that reconstructs the full phylogenetic lineage for any entry in taxonomy.dat.

In this blog, I include a Perl script that parses through taxonomy.dat, builds a hash of all the child-parent relationships, then re-parses the file, building the phylogenetic lineage of each organism using the child-parent hash that was built in the first pass.

This Perl script is provided "as is", without warranty of any kind, express or implied, including but not limited to the warranties of merchantability, fitness for a particular purpose and noninfringement. in no event shall the authors or copyright holders be liable for any claim, damages or other liability, whether in an action of contract, tort or otherwise, arising from, out of or in connection with the software or the use or other dealings in the software.

A copy of the Perl script is available at: http://www.julesberman.info/taxon.htm.

It takes under a minute to execute this script on a desktop computer running at 2.6 MHz with 512 MByte RAM. You may need this much RAM to provide memory for the hash (of child-parent relationships).

#!/usr/local/bin/perl
open(TAXO, "taxonomy.dat");
open(OUT, ">taxo.txt");
$/ = "//";
$line = " ";
while ($line ne "")
  {
  $line = <TAXO>;
  $line =~ /\nID +\: *([0-9]+) *\n/;
  $id_name = $1;
  $line =~ /\nPARENT ID +\: *([0-9]+) *\n/;
  $parent_id_name = $1;
  $parenthash{$id_name} = $parent_id_name;
  $line =~ /\nSCIENTIFIC NAME +\: *([^\n]+) *\n/;
  $scientific_name = $1;
  $namehash{$id_name} = $scientific_name;
  }
close(TAXO);
open(TAXO, "taxonomy.dat");
$line = " ";
while ($line ne "")
  {
  $line = <TAXO>;
  $getline = $line;
  $getline =~ s/\/\///o;
  print OUT $getline . "HIERARCHY\n";
  $line =~ /\nID +\: *([0-9]+) *\n/;
  $id_name = $1;
  for(1..30)
    {
    print OUT "$namehash{$id_name}\n";
    $id_name = $parenthash{$id_name};
    last if ($namehash{$id_name} eq "root");
    }
  print OUT "//";
  }
exit;

The script produces an output file, taxo.txt that exceeds 224 Megabytes in length. The output consists of the taxonomic entries from taxonomy.dat, along with the phylogentic lineage for each organism.

An sample ancestral lineage, for "maple trees" is:

Maple trees
ID                        : 4022
PARENT ID                 : 23672
RANK                      : genus
GC ID                     : 1
MGC ID                    : 1
SCIENTIFIC NAME           : Acer
GENBANK COMMON NAME       : maple trees
SYNONYM                   : Acer L.
HIERARCHY
Acer
Sapindaceae
Sapindales
eurosids II
rosids
core eudicotyledons
eudicotyledons
Magnoliophyta
Spermatophyta
Euphyllophyta
Tracheophyta
Embryophyta
Streptophytina
Streptophyta
Viridiplantae
Eukaryota
cellular organisms
//

A web site that produces the phylogeny of any entered species (in taxonomy.dat) is available at: http://www.julesberman.info/post.htm

key words: python programming language, phylogeny, taxonomy, taxa, taxon, ancestral lineage, classification, phylogenetics, python script, scripting language, species, phylum, genus

---

In June, 2014, my book, entitled Rare Diseases and Orphan Drugs: Keys to Understanding and Treating the Common Diseases was published by Elsevier. The book builds the argument that our best chance of curing the common diseases will come from studying and curing the rare diseases.

I urge you to read more about my book. There's a generous preview of the book at the Google Books site. If you like the book, please request your librarian to purchase a copy of this book for your library or reading room.

- Jules J. Berman, Ph.D., M.D.

Python script to extract phylogenetic lineages using the EBI taxonomy

In yesterday's blog, I discussed a Ruby script for creating phylogentic lineages for species included in taxonomy.dat. Here is the equivalent project, in Python.

Taxonomy.dat is a large, publicly available list of organisms. The file is available from the European Bioinformatics Institute (EBI). It contains over 400,000 species:

[A sample record in Taxonomy.dat]

ID : 350094
PARENT ID : 343736
RANK : species
GC ID : 1
MGC ID : 5
SCIENTIFIC NAME : Omalisus fontisbellaquei
MISSPELLING : Omalisus fontisbellaquaei
MISSPELLING : Omalisis fontisbellaguei
//

The taxonomy.dat file exceeds 100 megabytes in length.

The taxonomy.dat file is available for public download through anonymous ftp.

[ftp://ftp.ebi.ac.uk/pub/databases/taxonomy/]

Information about the taxonomy.dat file is found at:

[http://www.ebi.ac.uk/msd-srv/docs/dbdoc/ref_taxonomy.html]

Notice that the sample entry (above) provides an ID number for the entry organism, and for it's parent class. Since every organism and class has a parent, you can write a script that reconstructs the full phylogenetic lineage for any entry in taxonomy.dat.

In this blog, I include a Python script that parses through taxonomy.dat, builds a hash of all the child-parent relationships, then re-parses the file, building the phylogenetic lineage of each organism using the child-parent hash that was built in the first pass.

This Python script is provided "as is", without warranty of any kind, express or implied, including but not limited to the warranties of merchantability, fitness for a particular purpose and noninfringement. in no event shall the authors or copyright holders be liable for any claim, damages or other liability, whether in an action of contract, tort or otherwise, arising from, out of or in connection with the software or the use or other dealings in the software.

A copy of the Python script is available at: http://www.julesberman.info/taxon.htm.

It takes under a minute to execute this script on a desktop computer running at 2.6 MHz with 512 MByte RAM. You may need this much RAM to provide memory for the hash (of child-parent relationships).

#!/usr/local/bin/python
import re
intext = open("taxonomy.dat", "r")
outtext = open("taxo.txt", "w")
parenthash = {}
namehash = {}
cum_line = ""
childnumber = ""
parentnumber = ""
child_match = re.compile('ID\s+\:\s*(\d+)\s*')
parent_match = re.compile('PARENT ID\s+\:\s*(\d+)\s*')
name_match = re.compile('SCIENTIFIC NAME\s+\:\s*([^\n]+)\s*')
end_match = re.compile('\/\/')
for line in intext:
  p = end_match.search(line)
  if p:
    m = child_match.search(cum_line)
    if m:
      childnumber = m.group(1)
    x = parent_match.search(cum_line)
    if x:
      parentnumber = x.group(1)
    parenthash[childnumber] = parentnumber
    y = name_match.search(cum_line)
    if y:
      scientific_name = y.group(1)
    namehash[childnumber] = scientific_name
    #print childnumber + " " + namehash[childnumber] + " " + parenthash[childnumber]
    cum_line = ""
    continue
  else:
    cum_line = cum_line + line 
cum_line = ""
intext.close
intext = open("taxonomy.dat", "r")
for line in intext:
  p = end_match.search(line)
  if p:
    print>>outtext, cum_line + "HIERARCHY"
    z = child_match.search(cum_line)
    if z:
      id_name = z.group(1)
    for i in range(30):
      if namehash.has_key(id_name):
        print>>outtext, namehash[id_name]
      if parenthash.has_key(id_name):
        id_name = parenthash[id_name]
    print>>outtext, "//"
    cum_line = ""
    continue
  else:
    cum_line = cum_line + line 
cum_line = ""
exit

The script produces an output file, taxo.txt that exceeds 224 Megabytes in length. The output consists of the taxonomic entries from taxonomy.dat, along with the phylogentic lineage for each organism.

An sample ancestral lineage, for "bison" is:

ID                        : 9900
PARENT ID                 : 27592
RANK                      : genus
GC ID                     : 1
MGC ID                    : 2
SCIENTIFIC NAME           : Bison
HIERARCHY
Bison
Bovinae
Bovidae
Pecora
Ruminantia
Cetartiodactyla
Laurasiatheria
Eutheria
Theria
Mammalia
Amniota
Tetrapoda
Sarcopterygii
Euteleostomi
Teleostomi
Gnathostomata
Vertebrata
Craniata
Chordata
Deuterostomia
Coelomata
Bilateria
Eumetazoa
Metazoa
Fungi/Metazoa group
Eukaryota
cellular organisms

A web site that produces the phylogeny of any entered species (in taxonomy.dat) is available at: http://www.julesberman.info/post.htm

- Jules Berman

tags: python programming language, phylogeny, taxonomy, taxa, taxon, ancestral lineage, classification, phylogenetics, python script, scripting language, species, phylum, genus

---

Science is not a collection of facts. Science is what facts teach us; what we can learn about our universe, and ourselves, by deductive thinking. From observations of the night sky, made without the aid of telescopes, we can deduce that the universe is expanding, that the universe is not infinitely old, and why black holes exist. Without resorting to experimentation or mathematical analysis, we can deduce that gravity is a curvature in space-time, that the particles that compose light have no mass, that there is a theoretical limit to the number of different elements in the universe, and that the earth is billions of years old. Likewise, simple observations on animals tell us much about the migration of continents, the evolutionary relationships among classes of animals, why the nuclei of cells contain our genetic material, why certain animals are long-lived, why the gestation period of humans is 9 months, and why some diseases are rare and other diseases are common. In “Armchair Science”, the reader is confronted with 129 scientific mysteries, in cosmology, particle physics, chemistry, biology, and medicine. Beginning with simple observations, step-by-step analyses guide the reader toward solutions that are sometimes startling, and always entertaining. “Armchair Science” is written for general readers who are curious about science, and who want to sharpen their deductive skills.

Developmental Classification of Neoplasms now an RDF Ontology

I am publishing today the first ontology version of the Developmental Lineage Classification and Taxonomy of Neoplasms. It is available for download in several file versions.

The full ontology is a 10 Megabyte RDF file. Note that the file is so large that some browsers may not be able to open the entire file. On my computer, I had no trouble opening the file in my Internet Explorer browser, but the file was too large for my Mozilla browser.
http://www.julesberman.info/neordf.xml

The file was validated using the w3c validator service at http://www.w3.org/rdf/validator/, with a caveat. The full ontology file (10+ Mbytes) was too large for the validator, so I truncated the ontology, validated the truncated file (that contained all of the classes, subclasses, properties), and left out the repetitive list of terms. Then I took the entire file and validated it with an XML parser to verify that the file was well-formed. That really covers everything (RDF logic and XML structure).

The gzipped version of the RDF file (under 1 Megabyte).
http://www.julesberman.info/neorxml.gz

The flat file version, listing each term followed by its lineage (gzipped file).
http://www.julesberman.info/neoself.gz

The plain old XML version, with no RDF semantics (gzipped file). http://www.julesberman.info/neoclxml.gz

The ontology contains several parts:

1. The neoplasm classification proper (as illustrated in the schematic)



2. A listing of cancer terms that will probably never be entered into the proper classification (more about this later)

3. A listing of hyperplasias or hamartomas, some of which will be entered into the proper classification and others of which will remain in class Hyperplasia

4. A listing of precancer terms

5. A listing of syndromes associated with increased risk for cancer.

In this version, there are 5841 classified types of neoplasms and 130,503 terms representing the 5,841 types of neoplasms.

This represents the largest nomenclature of neoplasms in existence and, with today's publication, the largest formal ontology (in RDF syntax) of neoplasm names.

Over the next few weeks, I'll post additional blogs to further explain the RDF ontology files.

- Jules Berman

---

Science is not a collection of facts. Science is what facts teach us; what we can learn about our universe, and ourselves, by deductive thinking. From observations of the night sky, made without the aid of telescopes, we can deduce that the universe is expanding, that the universe is not infinitely old, and why black holes exist. Without resorting to experimentation or mathematical analysis, we can deduce that gravity is a curvature in space-time, that the particles that compose light have no mass, that there is a theoretical limit to the number of different elements in the universe, and that the earth is billions of years old. Likewise, simple observations on animals tell us much about the migration of continents, the evolutionary relationships among classes of animals, why the nuclei of cells contain our genetic material, why certain animals are long-lived, why the gestation period of humans is 9 months, and why some diseases are rare and other diseases are common. In “Armchair Science”, the reader is confronted with 129 scientific mysteries, in cosmology, particle physics, chemistry, biology, and medicine. Beginning with simple observations, step-by-step analyses guide the reader toward solutions that are sometimes startling, and always entertaining. “Armchair Science” is written for general readers who are curious about science, and who want to sharpen their deductive skills.