Data repurposing involves using old data in new ways, that were not foreseen by the people who originally collected
the data. Data repurposing comes in the following categories: (1) Using the preexisting data to ask and
answer questions that were not contemplated by the people who designed and collected the data; (2) Combining
preexisting data with additional data, of the same kind, to produce aggregate data that suits a new set of
questions that could not have been answered with any one of the component data sources; (3) Reanalyzing data
to validate assertions, theories, or conclusions drawn from the original studies; (4) Reanalyzing the original
data set using alternate or improved methods to attain outcomes of greater precision or reliability than the outcomes
produced in the original analysis; (5) Integrating heterogeneous data sets (ie, data sets with seemingly
unrelated types of information), for the purpose of answering questions or developing concepts that span diverse
scientific disciplines; (6) Finding subsets in a population once thought to be homogeneous; (7) Seeking
new relationships among data objects; (8) Creating, on-the-fly, novel data sets through data file linkages;
(9) Creating new concepts or ways of thinking about old concepts, based on a re-examination of data;
(10) Fine-tuning existing data models; and (11) Starting over and remodeling systems.
Berman JJ. Repurposing Legacy Data: Innovative Case Studies. Morgan Kaufmann, Waltham, MA, 2015.
-Jules Berman (copyrighted material)
key words: reanalysis, data science, secondary data, primary data, data integration, jules j berman
Showing posts with label data integration. Show all posts
Showing posts with label data integration. Show all posts
Sunday, January 24, 2016
Wednesday, January 20, 2016
TRIPLES: THE BASIC UNIT OF MEANING IN DATA SCIENCE
Data, by itself, has no meaning. It is the job of the data scientist to assign meaning to data, and this is
done with data objects, triples, and classifications. Our most familiar
data constructions (eg, spreadsheets, relational databases, flat-file records) convey meaning through
triples and data objects; we just don’t perceive them as such.
The three conditions for a meaningful assertion are:
Here are some examples of triples, as they might occur in a medical dataset:
- Jules Berman (copyrighted material)
key words: triple, meaning, informatics, computer science, data integration, heterogeneous data, jules j berman
The three conditions for a meaningful assertion are:
1. There is a specific data object about which the statement is made. 2. There is data that pertains to the specified object. 3. There is metadata that describes the dataSimply put, assertions have meaning whenever a pair of metadata and data (the descriptor for the data and the data itself) is assigned to a specific object. In the informatics field, assertions come in the form of so-called triples, consisting of the object, then the metadata, and then the data.
Here are some examples of triples, as they might occur in a medical dataset:
"Jules Berman" "blood glucose level" "85" "Mary Smith" "blood glucose level" "90" "Samuel Rice" "blood glucose level" "200" "Jules Berman" "eye color" "brown" "Mary Smith" "eye color" "blue" "Samuel Rice" "eye color" "green"Here are a few triples, as the might occur in a haberdasher’s dataset
"Juan Valdez" "hat size" "8" "Jules Berman" "hat size" "9" "Homer Simpson" "hat size" "9" "Homer Simpson" "hat_type" "bowler"We can combine the triples from a medical dataset and a habderdasher’s data set that apply to a common object:
"Jules Berman" "blood glucose level" "85" "Jules Berman" "eye color" "brown" "Jules Berman" "hat size" "9"Triples can port their meaning between different databases because they bind described data to an object. The portability of triples permits us to achieve data integration of heterogeneous data, and facilitates the design of software agents. Data integration involves merging related data objects, across diverse data sets. As it happens, if data supports introspection and data is organized as meaningful assertions (ie, as identified triples), then data integration is implicit (ie, an intrinsic property of the data). In essence, data integration is awarded to data scientists who apply data simplification techniques.
- Jules Berman (copyrighted material)
key words: triple, meaning, informatics, computer science, data integration, heterogeneous data, jules j berman
Wednesday, June 18, 2008
Interoperability efforts: please re-think
I'm sure that every reader of this blog has noticed that the word "interoperabiity" has become hackneyed, along with "standards" and "data integration".
What is wrong with having interoperability for hospitals, and interoperability for biomedical scientists, and interoperability for cancer researchers, and so on?
The problem is that interoperability should extend between the individual data domain. It is self-defeating to spend a lot of money (usually taxpayer's) on efforts that try to achieve information interoperability for one or another interest group.
Assuming that you can achieve any kind of interoperability, you'll still be faced with making the hospital data integrate with the medical research data, and the cancer research data, and so on.
Wouldn't it make a lot more sense to develop generalized methods for describing data, of any kind? That's what this blog is chiefly about (though I often digress). Anyone with data should use a general syntax for describing the data and for relating the data to other data.
The method for describing and relating data is RDF. The best way to advance data interoperability is to start with an RDF-literate workforce.
Last year I addressed a group of about 100 scientists who had convened to discuss image standards in pathology (my field). I asked for a show of hands for the number of people who had "heard of" RDF. Only two or three people raised their hands.
This is a really big problem. How can you achieve interoperability if nobody speaks the language of data specification, RDF?
I've posted, with Bill Moore, a primer in medical image specification, using RDF. It's not a bad place to start, if you're interested in the subject.
-Jules Berman
My book, Principles of Big Data: Preparing, Sharing, and Analyzing Complex Information was published in 2013 by Morgan Kaufmann.
I urge you to explore my book. Google books has prepared a generous preview of the book contents.
tags: big data, metadata, data preparation, data analytics, data repurposing, datamining, data mining, biomedical informatics, standards organizations, resource description framework, xml, data integration
What is wrong with having interoperability for hospitals, and interoperability for biomedical scientists, and interoperability for cancer researchers, and so on?
The problem is that interoperability should extend between the individual data domain. It is self-defeating to spend a lot of money (usually taxpayer's) on efforts that try to achieve information interoperability for one or another interest group.
Assuming that you can achieve any kind of interoperability, you'll still be faced with making the hospital data integrate with the medical research data, and the cancer research data, and so on.
Wouldn't it make a lot more sense to develop generalized methods for describing data, of any kind? That's what this blog is chiefly about (though I often digress). Anyone with data should use a general syntax for describing the data and for relating the data to other data.
The method for describing and relating data is RDF. The best way to advance data interoperability is to start with an RDF-literate workforce.
Last year I addressed a group of about 100 scientists who had convened to discuss image standards in pathology (my field). I asked for a show of hands for the number of people who had "heard of" RDF. Only two or three people raised their hands.
This is a really big problem. How can you achieve interoperability if nobody speaks the language of data specification, RDF?
I've posted, with Bill Moore, a primer in medical image specification, using RDF. It's not a bad place to start, if you're interested in the subject.
-Jules Berman
My book, Principles of Big Data: Preparing, Sharing, and Analyzing Complex Information was published in 2013 by Morgan Kaufmann.
I urge you to explore my book. Google books has prepared a generous preview of the book contents.
tags: big data, metadata, data preparation, data analytics, data repurposing, datamining, data mining, biomedical informatics, standards organizations, resource description framework, xml, data integration
Saturday, August 4, 2007
New Ontology research funding opportunity at NIH
On August 3, 2007, NIH announced a new funding opportunity.
http://grants.nih.gov/grants/guide/pa-files/PAR-07-425.html
Title: Data Ontologies for Biomedical Research (R01)
Release/Posted Date: August 3, 2007
Opening Date: December 18, 2007 (Earliest date an application may be submitted to Grants.gov)
Letters of Intent Receipt Date(s): December 18, 2007, August 18, 2008, December 22, 2009, and August 21, 2009 for the four separate receipt dates..
Application Submission/Receipt Date(s): January 18, 2008, September 18, 2008, January 21, 2009, and September 21, 2009
Expiration/Closing Date: September 22, 2009
Excerpt: "This FOA encourages the use, improvement, or development of techniques, tools, and better practices for integrating data sets by supporting projects that integrate existing data sets. Specifically, in this FOA, applicants should identify two (or more if they are very closely related) data sets (presumably contained in databases) that are not currently integrated. They should describe the vocabulary used in each database and should develop an ontology that will be suitable to join both data sets. The applicant must justify the importance in unifying these two data sets. NIH anticipates that once important data sets in a topical area have been unified that others in that area will adopt the emerging standard."
It seems that NIH is looking for relatively small grants, but this is probably the best approach. One of the advantages of ontologies is that they permit you do integrate data easily and cheaply. It wouldn't make any sense to develop this technology if every project was a mega-million dollar effort.
This announcement is another indication that major biomedical funding agencies understand the importance of specifying and integrating data. I hope that the research community responds with high-quality applications.
-Jules Berman
Science is not a collection of facts. Science is what facts teach us; what we can learn about our universe, and ourselves, by deductive thinking. From observations of the night sky, made without the aid of telescopes, we can deduce that the universe is expanding, that the universe is not infinitely old, and why black holes exist. Without resorting to experimentation or mathematical analysis, we can deduce that gravity is a curvature in space-time, that the particles that compose light have no mass, that there is a theoretical limit to the number of different elements in the universe, and that the earth is billions of years old. Likewise, simple observations on animals tell us much about the migration of continents, the evolutionary relationships among classes of animals, why the nuclei of cells contain our genetic material, why certain animals are long-lived, why the gestation period of humans is 9 months, and why some diseases are rare and other diseases are common. In “Armchair Science”, the reader is confronted with 129 scientific mysteries, in cosmology, particle physics, chemistry, biology, and medicine. Beginning with simple observations, step-by-step analyses guide the reader toward solutions that are sometimes startling, and always entertaining. “Armchair Science” is written for general readers who are curious about science, and who want to sharpen their deductive skills.
http://grants.nih.gov/grants/guide/pa-files/PAR-07-425.html
Title: Data Ontologies for Biomedical Research (R01)
Release/Posted Date: August 3, 2007
Opening Date: December 18, 2007 (Earliest date an application may be submitted to Grants.gov)
Letters of Intent Receipt Date(s): December 18, 2007, August 18, 2008, December 22, 2009, and August 21, 2009 for the four separate receipt dates..
Application Submission/Receipt Date(s): January 18, 2008, September 18, 2008, January 21, 2009, and September 21, 2009
Expiration/Closing Date: September 22, 2009
Excerpt: "This FOA encourages the use, improvement, or development of techniques, tools, and better practices for integrating data sets by supporting projects that integrate existing data sets. Specifically, in this FOA, applicants should identify two (or more if they are very closely related) data sets (presumably contained in databases) that are not currently integrated. They should describe the vocabulary used in each database and should develop an ontology that will be suitable to join both data sets. The applicant must justify the importance in unifying these two data sets. NIH anticipates that once important data sets in a topical area have been unified that others in that area will adopt the emerging standard."
It seems that NIH is looking for relatively small grants, but this is probably the best approach. One of the advantages of ontologies is that they permit you do integrate data easily and cheaply. It wouldn't make any sense to develop this technology if every project was a mega-million dollar effort.
This announcement is another indication that major biomedical funding agencies understand the importance of specifying and integrating data. I hope that the research community responds with high-quality applications.
-Jules Berman
Science is not a collection of facts. Science is what facts teach us; what we can learn about our universe, and ourselves, by deductive thinking. From observations of the night sky, made without the aid of telescopes, we can deduce that the universe is expanding, that the universe is not infinitely old, and why black holes exist. Without resorting to experimentation or mathematical analysis, we can deduce that gravity is a curvature in space-time, that the particles that compose light have no mass, that there is a theoretical limit to the number of different elements in the universe, and that the earth is billions of years old. Likewise, simple observations on animals tell us much about the migration of continents, the evolutionary relationships among classes of animals, why the nuclei of cells contain our genetic material, why certain animals are long-lived, why the gestation period of humans is 9 months, and why some diseases are rare and other diseases are common. In “Armchair Science”, the reader is confronted with 129 scientific mysteries, in cosmology, particle physics, chemistry, biology, and medicine. Beginning with simple observations, step-by-step analyses guide the reader toward solutions that are sometimes startling, and always entertaining. “Armchair Science” is written for general readers who are curious about science, and who want to sharpen their deductive skills.
Friday, May 25, 2007
Consequences of open access to clinical trial data
There was a very interesting development that was reported earlier this week, by Barry Meier in the May 23 New York Times.
Dr. Steven Nissen, a cardiologist at the Cleveland Clinic, had visited the Glaxo Wellcome clinical trials site, which contains data on drug trials. Nissen made an observation, based on Glaxo Wellcome's own data, that shed new light on the risks of cardiac side-effects with Avandia.
Without going into any of the specifics of this particular finding, I would just like to comment that Glaxo Wellcome made a valuable contribution to medicine simply by opening its clinical trial data to public review.
Research reports are always misleading. This is not because the authors of research reports are liars (although I'm sure that lying sometimes occurs), but because it is human nature to describe research findings in a way that supports a self-serving view of reality. That is why the primary data upon which research findings are based must always be open for public review.
By releasing de-identified clinical data to the public, the data can be reviewed and analyzed to yield fresh insights. Yes, sometimes public review may yield faults in the study that were overlooked by the original authors. Sometimes public review may yield conclusions that the original authors would rather not entertain. My personal hope is that more often than not, review of the data will yield new opportunities, new discoveries and new hypotheses that will benefit everyone.... and that will particularly benefit the original creators of the data. Ultimately, review of public datasets will support integrative studies that combine speficied data from many different studies to produce important results that were unanticipated by any of the original data creators.
-Jules Berman
In June, 2014, my book, entitled Rare Diseases and Orphan Drugs: Keys to Understanding and Treating the Common Diseases was published by Elsevier. The book builds the argument that our best chance of curing the common diseases will come from studying and curing the rare diseases.
I urge you to read more about my book. There's a generous preview of the book at the Google Books site.
tags: common disease, orphan disease, orphan drugs, genetics of disease, disease genetics, rules of disease biology, rare disease, pathology, clinical trial, data integration, data review, open access, trialists
Dr. Steven Nissen, a cardiologist at the Cleveland Clinic, had visited the Glaxo Wellcome clinical trials site, which contains data on drug trials. Nissen made an observation, based on Glaxo Wellcome's own data, that shed new light on the risks of cardiac side-effects with Avandia.
Without going into any of the specifics of this particular finding, I would just like to comment that Glaxo Wellcome made a valuable contribution to medicine simply by opening its clinical trial data to public review.
Research reports are always misleading. This is not because the authors of research reports are liars (although I'm sure that lying sometimes occurs), but because it is human nature to describe research findings in a way that supports a self-serving view of reality. That is why the primary data upon which research findings are based must always be open for public review.
By releasing de-identified clinical data to the public, the data can be reviewed and analyzed to yield fresh insights. Yes, sometimes public review may yield faults in the study that were overlooked by the original authors. Sometimes public review may yield conclusions that the original authors would rather not entertain. My personal hope is that more often than not, review of the data will yield new opportunities, new discoveries and new hypotheses that will benefit everyone.... and that will particularly benefit the original creators of the data. Ultimately, review of public datasets will support integrative studies that combine speficied data from many different studies to produce important results that were unanticipated by any of the original data creators.
-Jules Berman
In June, 2014, my book, entitled Rare Diseases and Orphan Drugs: Keys to Understanding and Treating the Common Diseases was published by Elsevier. The book builds the argument that our best chance of curing the common diseases will come from studying and curing the rare diseases.
I urge you to read more about my book. There's a generous preview of the book at the Google Books site.
tags: common disease, orphan disease, orphan drugs, genetics of disease, disease genetics, rules of disease biology, rare disease, pathology, clinical trial, data integration, data review, open access, trialists
Tuesday, April 3, 2007
More Introduction to RDF
As discussed in an earlier post, RDF (Resource Description Framework) is a formal method for describing specified data objects with paired metadata and data.
It is important to understand that in informatics, assertions only have meaning when a pair of metadata and data (the descriptor for the data and the data itself) is assigned to a specific subject.
The "triples" that form the basis of RDF specifications are: Specified subject then metadata then data
Examples of triples that might be found in a medical dataset:
“Jules Berman” “blood glucose level” “85”
“Mary Smith” “blood glucose level” “90”
“Samuel Rice” “blood glucose level" "200"
“Jules Berman” “eye color” “brown”
“Mary Smith” “eye color” “blue”
“Samuel Rice” “eye color" "green"
Some triples found in a haberdasher's dataset
“Juan Valdez” “hat size” “8”
“Jules Berman” “hat size” “9”
“Homer Simpson” “hat size” “9”
“Homer Simpson” “hat_type” “bowler”
Triples collected from both datasets whose subject is "Jules Berman"
“Jules Berman” “blood glucose level” “85”
“Jules Berman” “eye color” “brown”
“Jules Berman” “hat size” “9”
This is a simple example of data integration over heterogeneous datasets!
Triples can port their meaning between different databases because they bind described data to a specified subject. This supports data integration of heterogeneous data and facilitates the design of software agents. A software agent, as used here, is a program that can interrogate multiple RDF documents on the web, initiating its own actions based on inferences yielded from retrieved triples.
RDF (Resource Description Framework) is a syntax for writing computer-parsable triples. For RDF to serve as a general method for describing data objects, we need to answer the following four questions:.
1. How does the triple convey the unique identity of its subject? In the triple, “Jules Berman” “blood glucose level” “85”, The name "Jules Berman" is not unique and may apply to several different people.
2. How do we convey the meaning of metadata terms? Perhaps one person's definition of a metadata term is different from another person's. For example, is "hat size" the diameter of the hat, or the distance from ear to ear on the person who is intended to wear the hat, or a digit selected from a pre-defined scale?
3. How can we constrain the values described by metadata to a specific datatype? Can a person have an eye color of 8? Can a person have an eye color of "chartreuse"?
4. How can we indicate that a unique object is a member of a class and can be described by metadata shared by all the members of a class?
In subsequent blog posts, we'll examine how RDF provides answers to these four questions.
-Jules Berman tags: data integration, meaning, rdf, specifications, standards, triples, science
Science is not a collection of facts. Science is what facts teach us; what we can learn about our universe, and ourselves, by deductive thinking. From observations of the night sky, made without the aid of telescopes, we can deduce that the universe is expanding, that the universe is not infinitely old, and why black holes exist. Without resorting to experimentation or mathematical analysis, we can deduce that gravity is a curvature in space-time, that the particles that compose light have no mass, that there is a theoretical limit to the number of different elements in the universe, and that the earth is billions of years old. Likewise, simple observations on animals tell us much about the migration of continents, the evolutionary relationships among classes of animals, why the nuclei of cells contain our genetic material, why certain animals are long-lived, why the gestation period of humans is 9 months, and why some diseases are rare and other diseases are common. In “Armchair Science”, the reader is confronted with 129 scientific mysteries, in cosmology, particle physics, chemistry, biology, and medicine. Beginning with simple observations, step-by-step analyses guide the reader toward solutions that are sometimes startling, and always entertaining. “Armchair Science” is written for general readers who are curious about science, and who want to sharpen their deductive skills.
It is important to understand that in informatics, assertions only have meaning when a pair of metadata and data (the descriptor for the data and the data itself) is assigned to a specific subject.
The "triples" that form the basis of RDF specifications are: Specified subject then metadata then data
Examples of triples that might be found in a medical dataset:
“Jules Berman” “blood glucose level” “85”
“Mary Smith” “blood glucose level” “90”
“Samuel Rice” “blood glucose level" "200"
“Jules Berman” “eye color” “brown”
“Mary Smith” “eye color” “blue”
“Samuel Rice” “eye color" "green"
Some triples found in a haberdasher's dataset
“Juan Valdez” “hat size” “8”
“Jules Berman” “hat size” “9”
“Homer Simpson” “hat size” “9”
“Homer Simpson” “hat_type” “bowler”
Triples collected from both datasets whose subject is "Jules Berman"
“Jules Berman” “blood glucose level” “85”
“Jules Berman” “eye color” “brown”
“Jules Berman” “hat size” “9”
This is a simple example of data integration over heterogeneous datasets!
Triples can port their meaning between different databases because they bind described data to a specified subject. This supports data integration of heterogeneous data and facilitates the design of software agents. A software agent, as used here, is a program that can interrogate multiple RDF documents on the web, initiating its own actions based on inferences yielded from retrieved triples.
RDF (Resource Description Framework) is a syntax for writing computer-parsable triples. For RDF to serve as a general method for describing data objects, we need to answer the following four questions:.
1. How does the triple convey the unique identity of its subject? In the triple, “Jules Berman” “blood glucose level” “85”, The name "Jules Berman" is not unique and may apply to several different people.
2. How do we convey the meaning of metadata terms? Perhaps one person's definition of a metadata term is different from another person's. For example, is "hat size" the diameter of the hat, or the distance from ear to ear on the person who is intended to wear the hat, or a digit selected from a pre-defined scale?
3. How can we constrain the values described by metadata to a specific datatype? Can a person have an eye color of 8? Can a person have an eye color of "chartreuse"?
4. How can we indicate that a unique object is a member of a class and can be described by metadata shared by all the members of a class?
In subsequent blog posts, we'll examine how RDF provides answers to these four questions.
-Jules Berman tags: data integration, meaning, rdf, specifications, standards, triples, science
Science is not a collection of facts. Science is what facts teach us; what we can learn about our universe, and ourselves, by deductive thinking. From observations of the night sky, made without the aid of telescopes, we can deduce that the universe is expanding, that the universe is not infinitely old, and why black holes exist. Without resorting to experimentation or mathematical analysis, we can deduce that gravity is a curvature in space-time, that the particles that compose light have no mass, that there is a theoretical limit to the number of different elements in the universe, and that the earth is billions of years old. Likewise, simple observations on animals tell us much about the migration of continents, the evolutionary relationships among classes of animals, why the nuclei of cells contain our genetic material, why certain animals are long-lived, why the gestation period of humans is 9 months, and why some diseases are rare and other diseases are common. In “Armchair Science”, the reader is confronted with 129 scientific mysteries, in cosmology, particle physics, chemistry, biology, and medicine. Beginning with simple observations, step-by-step analyses guide the reader toward solutions that are sometimes startling, and always entertaining. “Armchair Science” is written for general readers who are curious about science, and who want to sharpen their deductive skills.
Subscribe to:
Posts (Atom)