Showing posts with label perl programming. Show all posts
Showing posts with label perl programming. Show all posts

Tuesday, August 3, 2010

Image montage with Perl Ruby Python

On occasion, you might want to create a single image from a list of individual images. The easiest way of making such an image is with ImageMagick.

ImageMagick is a free, open source, image application. Information is available at:

http://www.imagemagick.org/Usage/

If you use Windows, a compiled binary download is available:

Perl, Python, and Ruby each have their own interfaces to ImageMagick.

Today's blog provides equivalent implementations of the ImageMagick montage feature, in three languages.

Perl:

#!/usr/bin/perl
use Image::Magick;
my $image = Image::Magick->new;
my $montage = Image::Magick->new;
my $status = $image -> Read("mach_cov.jpg",
"071.jpg", "072.jpg", "073.jpg", "075.jpg",
"076.jpg", "077.jpg", "079.jpg");
$image -> Scale('200x200');
print STDERR $status;
$montage = $image -> Montage(background => "Blue",
borderwidth => "5", geometry => '+5+5',
tile => "4x2");
$montage -> Write('jpg:orig.jpg');
system("imdisplay orig.jpg");
exit;

Python:

#!/usr/bin/python
import os
os.system("montage mach_cov.jpg \
071.jpg 072.jpg 073.jpg 075.jpg \
076.jpg 077.jpg 079.jpg \
-background #0000ff -geometry 200x200+5+5 \
-borderwidth 5 -tile 4x2 orig.gif")
os.system("imdisplay orig.gif");
exit

The "\" appearing at the end of lines is the DOS command-line continuation character.

Ruby:

#!/usr/local/bin/ruby
require 'RMagick'
include Magick
orig = Magick::ImageList.new(
"mach_cov.jpg", "071.jpg", "072.jpg",
"073.jpg", "075.jpg", "076.jpg",
"077.jpg", "079.jpg")
new_img = orig.montage() {
self.background_color='blue';
self.border_color='red';
self.tile='4x2'
self.geometry='200x200+5+15'
self.border_width=5}
new_img.write("orig.gif")
system("imdisplay orig.gif")
exit

The input images are the 8 book covers that appear on the right hand side of this blog. Obviously, you'll substitute any number of images from your own collection. You'll also need to provide a an appropriate tiling geometry for the number of images you use (I used 4x2, four across and 2 down) for this example.

The output looks like this:


Each of the scripts calls provides the list of images to be included in the output image, and provides the montage method with attributes for the image display (i.e., background collor, image sizes, image border, space between images, tiling geometry).

In the case of the Perl script, the external module Image::Magick is called (available from ActiveState). In the case of Ruby, the RMagick gem is used. Users of these languages should known how to acquire these modules.

In the case of Python, a system call is made to the installed ImageMagick application. I had a little problem getting the Python script to work because the system call to the "montage" executable apparently executed some other program of the same name. This problem was skirted when I simply visited the subdirectory that held the ImageMagick files, and copied the montage.exe file over to the same subdirectory that held my Python script. When the python script was interpreted, the system used the montage.exe file in the same subdirectory as the script.

The montage method is explained in detail on the following web page:

http://www.imagemagick.org/Usage/montage/

Many of the basic algorithms of Medical Informatics, with equivalent implementations in Perl, Python, and Ruby are provided in my book, Methods in Medical Informatics: Fundamentals of Healthcare Programming in Perl, Python, and Ruby, which will be published next month by CRC Press.

- © 2010 Jules J. Berman
tags:image files, perl programming, Python programming, Ruby programming
Science is not a collection of facts. Science is what facts teach us; what we can learn about our universe, and ourselves, by deductive thinking. From observations of the night sky, made without the aid of telescopes, we can deduce that the universe is expanding, that the universe is not infinitely old, and why black holes exist. Without resorting to experimentation or mathematical analysis, we can deduce that gravity is a curvature in space-time, that the particles that compose light have no mass, that there is a theoretical limit to the number of different elements in the universe, and that the earth is billions of years old. Likewise, simple observations on animals tell us much about the migration of continents, the evolutionary relationships among classes of animals, why the nuclei of cells contain our genetic material, why certain animals are long-lived, why the gestation period of humans is 9 months, and why some diseases are rare and other diseases are common. In “Armchair Science”, the reader is confronted with 129 scientific mysteries, in cosmology, particle physics, chemistry, biology, and medicine. Beginning with simple observations, step-by-step analyses guide the reader toward solutions that are sometimes startling, and always entertaining. “Armchair Science” is written for general readers who are curious about science, and who want to sharpen their deductive skills.

Friday, July 23, 2010

New book coming in September

If you've noticed a decline in my blogging of late, it's because I've been putting the finishing touches on my new book, Methods in Medical Informatics: Fundamentals of Healthcare Programming in Perl, Python, and Ruby, which will be published in September in the CRC Press series in Mathematical and Computational Biology.

This book is a departure from my earlier language-specific works. The new book stresses descriptions of medical informatics algorithms, all presented in the following format:

1) Each method has a background discussion, explaining how the method is used by healthcare professionals.

2) This is followed by a narrative description of each of the steps of the algorithm.

3) Equivalent implementations of the algorithm are provided in Perl, Python and Ruby. Scripts draw data from freely available datasets (OMIM, SEER, CDC, Census Bureau, MeSH, and PubMed, Taxonomy).

4) Then comes an analysis of the output of the algorithm.

At the end of every chapter, a list of exercises provides students with short projects, that they can develop in the language of their own choice.

There are about 70 algorithms or methods provided in the book, and with a few exceptions (some of the introductory-level methods), these algorithms have not appeared in any of my prior publications.

The unique offering of this book is that students can master the fundamental methods and algorithms of medical informatics, in their preferred programming language. Instructors can concentrate on teaching useful algorithms, without wasting class time on language-specific programming instruction.

The book is divided into four parts: Part I. Building blocks of medical informatics; Part II. Free, publicly available medical data resources; Part III. The common tasks of medical informatics, and Part IV. Medical Discovery.

The purpose of Part I, Building blocks of medical informatics, is to introduce the basic computational subroutines that will be used in more complex scripts later in the book. Everyone who reads the book is expected to have a basic understanding of either Perl, Python, or Ruby. Readers can skip over the parts written for the other languages. Over the years, students will find it valuable to re-read the text, this time paying attention to the languages they ignored in the first pass.

In Part II, Free, publicly available medical data resources, students are shown how to use the U.S. government biomedical datasets. The chapters in Part II explain the intended uses of these datasets, how the data sets are organized, and how students can retrieve and analyze the data.

Part III, The common tasks of medical informatics, covers some of the computational methods of biomedical informatics, including autocoding, data scrubbing, and data de-identification.

Part IV, Medical discovery, provides examples of the kinds of questions that biomedical scientists can ask and answer with public data and open source programming languages. Students will learn techniques for combining heterogeneous data sources, measuring and visualizing trends in complex data, and testing new hypotheses. The majority of research and development projects that use biomedical data can be developed using the methods described in Part IV.

This book is written specifically as a textbook for courses in medical informatics. Instructors should appreciate this book, because it frees them to teach medical informatics, without wasting time teaching basic programming skills. The included scripts are for the students, who need code samples, from a familiar language, that they can use throughout their careers. The instructors will teach the algorithms (not the language implementations). This strategy allows instructors to teach a big subject (computational medical informatics), in a small amount of time (one academic course).

The book is designed to eliminate the inequities that result when an instructor imposes his choice [of programming language] on all of his students. Students who are trained in an alternate language will resent taking a course in an unfamiliar language. These students will be at a disadvantage compared to other students who happen to be trained in the course-book language. When a professor selects a computer language, he's usually got to start the course with the basics of the selected language, and this instruction can takes weeks of time away from the subject of the course [in this case, medical informatics]. This is just lame.

Creative Writing courses do not waste time teaching people about pens and pencils. Informatics courses should not be waste time teaching programming skills. When a medical informatics instructor tells everyone in the class to use Python, it's equivalent to a writing instructor telling everyone in the class to use a Papermate pen. A good course in medical informatics should expect students to have programming skills, but should not be focused on any particular programming language. Instructors should focus on the tasks, issues, and questions that comprise the core activities of professionals in the field. By providing a discussion of the importance and utility of each algorithm and method, instructors can teach the fundamentals of medical informatics, without wasting time teaching programming.

All of the data used in this book are free and publicly available. Most of the data comes from U.S. government sources, providing hundreds of gigabytes of high quality, curated biomedical data to a global community of scientists, healthcare experts, clinicians, nurses, and students. Every student should become familiar with these data sources, and understand their medical value. This book provides instructions for downloading all of the data sources discussed.

One of the most surprising features of medical informatics is that scripts capable of parsing large data sets, very quickly, can be quite short. By focusing on methods, not full-scale applications, scripts are kept short, concise, and easy to learn. In many cases, the scripts that implement an algorithm are actually shorter than the step-by-step description that precedes the implementation. Students will find that most of their future data analysis efforts can be accomplished by modifying and combining the scripts provided with this book.

-© 2010 Jules J. Berman


Science is not a collection of facts. Science is what facts teach us; what we can learn about our universe, and ourselves, by deductive thinking. From observations of the night sky, made without the aid of telescopes, we can deduce that the universe is expanding, that the universe is not infinitely old, and why black holes exist. Without resorting to experimentation or mathematical analysis, we can deduce that gravity is a curvature in space-time, that the particles that compose light have no mass, that there is a theoretical limit to the number of different elements in the universe, and that the earth is billions of years old. Likewise, simple observations on animals tell us much about the migration of continents, the evolutionary relationships among classes of animals, why the nuclei of cells contain our genetic material, why certain animals are long-lived, why the gestation period of humans is 9 months, and why some diseases are rare and other diseases are common. In “Armchair Science”, the reader is confronted with 129 scientific mysteries, in cosmology, particle physics, chemistry, biology, and medicine. Beginning with simple observations, step-by-step analyses guide the reader toward solutions that are sometimes startling, and always entertaining. “Armchair Science” is written for general readers who are curious about science, and who want to sharpen their deductive skills.

Sunday, January 24, 2010

Grabbing, Testing Web Links

Once more, I'm interrupting a series of blogs on the topic of complexity to provide a pointer to a recently constructed web page showing how to grab web pages and test web links, using Perl, Python, or Ruby.

Perl, Python and Ruby have their own external modules for HTTP transactions. Each language's module has its own peculiar syntax. Still, the basic operation is the same: your script initiates an HTTP request for a web file at a specific network address (the URL, or Uniform Resource Locator). A response is received determining if the page is available (equivalent to testing a link). With a little effort, you can modify the provided scripts to collect and examine a large number of web pages. With a little more effort, you can write your own spider software that searches for web addresses, iteratively collecting information from links within web pages.

The article is at:

http://www.julesberman.info/factoids/url_get.htm


© 2010 Jules J. Berman

key words: html, http, hypertext transfer protocol, testing link, ruby programming, perl programming, python programming, bioinformatics, valid web page, web page is available, good http request, valid http request testing if web page exists, testing web links, jules berman Ph.D., M.D.
Science is not a collection of facts. Science is what facts teach us; what we can learn about our universe, and ourselves, by deductive thinking. From observations of the night sky, made without the aid of telescopes, we can deduce that the universe is expanding, that the universe is not infinitely old, and why black holes exist. Without resorting to experimentation or mathematical analysis, we can deduce that gravity is a curvature in space-time, that the particles that compose light have no mass, that there is a theoretical limit to the number of different elements in the universe, and that the earth is billions of years old. Likewise, simple observations on animals tell us much about the migration of continents, the evolutionary relationships among classes of animals, why the nuclei of cells contain our genetic material, why certain animals are long-lived, why the gestation period of humans is 9 months, and why some diseases are rare and other diseases are common. In “Armchair Science”, the reader is confronted with 129 scientific mysteries, in cosmology, particle physics, chemistry, biology, and medicine. Beginning with simple observations, step-by-step analyses guide the reader toward solutions that are sometimes startling, and always entertaining. “Armchair Science” is written for general readers who are curious about science, and who want to sharpen their deductive skills.

Tuesday, December 30, 2008

Why I write scripts in Perl, Ruby and Python

Readers of this blog know that I often include equivalent scripts in Perl, Ruby and Python, for the biomedical informatics projects that I post.

It must be tedious to encounter a blog that devotes long stretches of space to scripts written in three different languages. I can imagine the groans coming from non-programmers when they see a long post full of code, while all they're interested in is the biomedical problem described in the first paragraph of the post.

I have reasons for including these scripts:

1. I want to reach all of the people who are interested in the biomedical problems discussed in my posts. Most of these problems are solved with informatics techniques. The techniques often involve a little bit of programming knowledge. Most biomedical programming is done with Perl, Ruby or Python. These are free, open source and cross-platform languages with active user communities and with abundant instructional material on the internet. If I want to attract the maximum number of readers, I've got to include solutions in all three languages.

2. I want to show readers that powerful scripts, that solve important biomedical problems, can be written in a few lines of code, regardless of the language used. Perl, Ruby and Python can all be used to write equivalent programs. For length, clarity, and speed of execution, there really isn't much difference among the major scripting languages. Biomedical scripts tend to use a few favorite commands (open a file, parse the file line by line, extract something from the lines of the file, do some sort of transformation on the extracted data, write the results to another file). These commands can be learned in a few hours.

3. I want to emphasize in all of my blogs that the difficult part in any informatics project is developing your question (i.e., asking a smart, important, and solvable question), and understanding the substance and limitations of the available data sources. Writing scripts is the easiest and most enjoyable part of the exercise.

- © 2008 Jules J. Berman
Science is not a collection of facts. Science is what facts teach us; what we can learn about our universe, and ourselves, by deductive thinking. From observations of the night sky, made without the aid of telescopes, we can deduce that the universe is expanding, that the universe is not infinitely old, and why black holes exist. Without resorting to experimentation or mathematical analysis, we can deduce that gravity is a curvature in space-time, that the particles that compose light have no mass, that there is a theoretical limit to the number of different elements in the universe, and that the earth is billions of years old. Likewise, simple observations on animals tell us much about the migration of continents, the evolutionary relationships among classes of animals, why the nuclei of cells contain our genetic material, why certain animals are long-lived, why the gestation period of humans is 9 months, and why some diseases are rare and other diseases are common. In “Armchair Science”, the reader is confronted with 129 scientific mysteries, in cosmology, particle physics, chemistry, biology, and medicine. Beginning with simple observations, step-by-step analyses guide the reader toward solutions that are sometimes startling, and always entertaining. “Armchair Science” is written for general readers who are curious about science, and who want to sharpen their deductive skills.

Monday, December 29, 2008

Perl, Ruby and Python scripts for CDC public use mortality file parsing

I have been interested in knowing whether sickle cell incidence is decreasing in the U.S. population. Despite Pubmed and web searches, I have not been able to find a single data source on the subject. In a prior post, I referred to one of my Perl scripts that
parsed through CDC public use records (about 5 Gigabytes of raw data). The results seemed to suggest that the incidence of sickle cell anemia in the U.S. may be increasing.

For those interested, here are Perl, Ruby and Python versions of the script that parses through the CDC public use mortality files for the years 1996, 1999, 2002, and 2004, to produce the number of occurrences and the rates for sickle cell disease cases, from death certificate data, for those years.

Please refer to the prior post for discussion of the data.

Please refer to my document describing uses for the CDC public use mortality files for general instructions on acquiring and analyzing the de-identified U.S. death certificate data.

#!/usr/local/bin/perl
@filearray = qw(mort96us.dat mort99us.dat mort02us.dat mort04us.dat);
foreach $file (@filearray)
{
open (ICD, $file);
$line = " ";
$popcount = 0;
$counter = 0;
while ($line ne "")
{
$line = <ICD>;
$codesection = substr($line,448,140) if ($file eq $filearray[0]);
$codesection = substr($line,161,140) if ($file eq $filearray[1]);
$codesection = substr($line,162,140) if ($file eq $filearray[2]);
$codesection = substr($line,164,140) if ($file eq $filearray[3]);
$popcount++;
if ($codesection =~ /D57/i)
{
$counter++;
}
}
close ICD;
$rate = $counter / $popcount;
$rate = substr((100000 * $rate),0,5);
print "\n\nRecords listing sickle cell is $counter in $file file";
print "\nSickle cell rate per 100,000 records is $rate in $file file";
}
exit;

#!/usr/local/bin/ruby
filearray = Array.new
filearray = "mort96us.dat mort99us.dat mort02us.dat mort04us.dat".split
filearray.each do
|file|
text = File.open(file, "r")
counter = 0; popcount = 0;
text.each_line do
|line|
codesection = line[448,140] if (file == filearray.fetch(0))
codesection = line[161,140] if (file == filearray.fetch(1))
codesection = line[162,140] if (file == filearray.fetch(2))
codesection = line[164,140] if (file == filearray.fetch(3))
popcount = popcount +1
counter = (counter + 1) if (codesection =~ /D57/i)
end
text.close
rate = ((counter.to_f / popcount.to_f) * 100000).to_s[0,5]
puts "\nRecords listing sickle cell is #{counter} in #{file} file"
puts "Sickle cell rate per 100,000 records is #{rate} in #{file} file"
end
exit

#!/usr/local/bin/python
import re
sickle_match = re.compile('D57')
lst = ("mort96us.dat","mort99us.dat","mort02us.dat","mort04us.dat")
for file in lst:
intext = open(file, "r")
popcount = 0
counter = 0
codesection = ""
for line in intext:
if file == lst[0]:
codesection = line[448:588]
if file == lst[1]:
codesection = line[161:301]
if file == lst[2]:
codesection = line[162:302]
if file == lst[3]:
codesection = line[164:304]
popcount = popcount + 1
p = sickle_match.search(codesection)
if p:
counter = counter + 1
intext.close
rate = float(counter) / float(popcount) * 100000
rate = str(rate)
rate = rate[0:5]
print ('\n\nRecords listing sickle cell is ')
print (str(counter) + ' in ' + file + ' file')
print ('\nSickle cell rate per 100,000 records is ')
print(str(rate) + ' in ' + file + ' file')
exit

© 2008 Jules Berman

As with all of my scripts, lists, web sites, and blog entries, the following disclaimer applies. This material is provided by its creator, Jules J. Berman, "as is", without warranty of any kind, expressed or implied, including but not limited to the warranties of merchantability, fitness for a particular purpose and noninfringement. In no event shall the author or copyright holder be liable for any claim, damages or other liability, whether in an action of contract, tort or otherwise, arising from, out of or in connection with the material or the use or other dealings.

In June, 2014, my book, entitled Rare Diseases and Orphan Drugs: Keys to Understanding and Treating the Common Diseases was published by Elsevier. The book builds the argument that our best chance of curing the common diseases will come from studying and curing the rare diseases.



I urge you to read more about my book. There's a generous preview of the book at the Google Books site. If you like the book, please request your librarian to purchase a copy of this book for your library or reading room.

tags: common disease, orphan disease, orphan drugs, rare disease, disease genetics, sickle cell anemia, genetic counseling, public health, centers for disease control and prevention, international classification of diseases, cause of death, causes of death, epidemiology, Perl programming, Ruby programming, Python programming

Friday, April 18, 2008

MeSH (Medical Subject Headings) more complex than simple "Trees"

MeSH (Medical Subject Headings) is a wonderful nomenclature of medical terms available from the U.S. National Library of Medicine.

The download site is:

http://www.nlm.nih.gov/mesh/filelist.html


MeSH is one of the greatest gifts provided by the U.S. National Library of Medicine and can be used freely for a variety of projects involving indexing, tagging, searching, retrieving, coding, analyzing, merging, and sharing biomedical text. In my opinion, there are many projects that rely on commercial and legally encumbered nomenclatures that would be better served by MeSH.

My only quibble with MeSH is that it is incorrectly described as a Tree structure.

Here is the official word (from the NLM website) on MeSH Trees from: http://www.nlm.nih.gov/mesh/intro_trees2007.html

"Because of the branching structure of the hierarchies, these lists are sometimes referred to as "trees". Each MeSH descriptor appears in at least one place in the trees, and may appear in as many additional places as may be appropriate. Those who index articles or catalog books are instructed to find and use the most specific MeSH descriptor that is available to represent each indexable concept."

When you look at individual entries in MeSH, you find that a single entry may be assigned multiple MeSH numbers.

For example, the MeSH term, "Family" is assigned two MeSH numbers,

MN = F01.829.263
MN = I01.880.225

The parent "number" for any MeSH entry is found by removing the last set of decimal demarcated digits.

For example:
F01.829.263 MeSH name, Family
F01.829 MeSH name, Plychology, Social
F01 MeSH Name, Behavior and Behavior Mechanisms

For each MeSH number, there is a separate hierarchy.

It is tempting to think of each hierarchy for each number as a tree (then MeSH could be envisioned as a dense forest), but each parent term could be assigned multiple MeSH numbers, each producing a multi-branching hierarchy.

Because each MeSH term (including the ancestral terms for a MeSh term) may be assigned multiple MeSH numbers, each with its own hierarchy, the MeSH data structure is more accurately thought of as a complex ontology, with terms existing in multiple classes, with specified relationships among any class and its parent classes.

The tree metaphor breaks down because branches and nodes within a branch can be connected to other branches and to other nodes. Trees do not do this kind of thing.

It is possible to write a script that parses through every MeSH entry, finds all of the MeSH numbers for the entry, determines the parent terms for the MeSH numbers, determines all of the alternate MeSH numbers for the parent terms, then finds all of the grandparent terms for all of the parent terms, etc., until all of the hierarchical terms for the term are found.

Here is the Perl script. This Perl script is provided "as is", by its creator, Jules J. Berman, without warranty of any kind, express or implied, including but not limited to the warranties of merchantability, fitness for a particular purpose and noninfringement. in no event shall the authors or copyright holders be liable for any claim, damages or other liability, whether in an action of contract, tort or otherwise, arising from, out of or in connection with the software or the use or other dealings in the software.

#!/usr/local/bin/perl
open(MESH, "D2007.BIN"); #the file name for the raw ascii MeSH version
open(OUT, ">mesh.out");
$/ = "\n\n\*NEWRECORD\n";
$line = " ";
@cumlist;
%numberhash;
%namehash;
while ($line ne "")
{
my $numbers = "";
$line = <MESH>;
$line =~ /\nMH = ([^\n]+)\n/;
$name = $1;
while ($line =~ m/\nMN ?= ?([^\n]+)(?=\n)/mg)
{
$number = $1;
$number =~ s/^ *//o;
$number =~ s/ *$//o;
$number =~ s/ +/ /;
$numberhash{$number} = $name;
$numbers = $numbers . " " . $number;
}
$numbers =~ s/^ *//o;
$numbers =~ s/ *$//o;
$numbers =~ s/ +/ /o;
$namehash{$name} = $numbers;
}
close(MESH);
while((my $key, my $value) = each (%namehash))
{
@cumlist = ("");
print OUT "\nTERM LINEAGE FOR " . uc($key) . "\n";
my @valuelist = split(/ /,$value);
@cumlist = (@cumlist, @valuelist);
&splitlist(@cumlist);
for(1..30)
{
@cumlist = grep { $marked{$_}++; $marked{$_} == 1; }
@cumlist;
undef(%marked);
&allmeshnums(@cumlist);
@cumlist = grep { $marked{$_}++; $marked{$_} == 1; }
@cumlist;
undef(%marked);
&splitlist(@cumlist);
}
@cumlist = grep { $marked{$_}++; $marked{$_} == 1; }
@cumlist;
undef(%marked);
foreach my $thing (@cumlist)
{
print OUT "$thing $numberhash{$thing}\n";
}
}
sub splitlist()
{
@valuelist = @_;
foreach my $meshno (@valuelist)
{
for(1..30)
{
if ($meshno =~ /\.[0-9]+$/)
{
$meshno = $`;
push(@cumlist, $meshno);
}
else
{
last;
}
}
}
}
sub allmeshnums()
{
@meshnumber = @_;
foreach my $thing (@meshnumber)
{
my $name = $numberhash{$thing};
my $value = $namehash{$name};
my @valuelist = split(/ /,$value);
@cumlist = (@valuelist, @cumlist);
}
}
exit;

The output file, mesh.out is over 9 megabytes in length.

Here is an example of one entry, in the output file, mesh.out

TERM LINEAGE FOR GIANT CELLS, FOREIGN-BODY
A11.118.637 Leukocytes
A15.145.229.637 Leukocytes
A15.382.490 Leukocytes
A11.118.637.555 Leukocytes, Mononuclear
A15.145.229.637.555 Leukocytes, Mononuclear
A15.382.490.555 Leukocytes, Mononuclear
A15.378 Hematopoietic System
A11.148 Bone Marrow Cells
A15.378.316 Bone Marrow Cells
A12.207.152 Blood
A15.145 Blood
A11.118 Blood Cells
A15.145.229 Blood Cells
A11.329.372.376 Giant Cells, Foreign-Body
A11.502.376 Giant Cells, Foreign-Body
A11.627.624.480.376 Giant Cells, Foreign-Body
A11.733.397.376 Giant Cells, Foreign-Body
A15.382.680.397.376 Giant Cells, Foreign-Body
A15.382.812.522.376 Giant Cells, Foreign-Body
A11.329 Connective Tissue Cells
A11 Cells
A11.502 Giant Cells
A11.118.637.555.652 Monocytes
A11.148.580 Monocytes
A11.627.624 Monocytes
A11.733.547 Monocytes
A15.145.229.637.555.652 Monocytes
A15.378.316.580 Monocytes
A15.382.490.555.652 Monocytes
A15.382.680.547 Monocytes
A15.382.812.547 Monocytes
A11.627 Myeloid Cells
A11.733 Phagocytes
A15.382.680 Phagocytes
A15.382 Immune System
A15 Hemic and Immune Systems
A11.329.372 Macrophages
A11.627.624.480 Macrophages
A11.733.397 Macrophages
A15.382.680.397 Macrophages
A15.382.812.522 Macrophages
A15.382.812 Reticuloendothelial System
A12.207 Body Fluids
A12 Fluids and Secretions

When we examine the multi-lineage ancestry of "Foreign body giant cells" we see that MeSH is not a tree hierarchy. This means that the MeSH data structure is highly complex and requires some computational know-how to fully explore all the term relationships.

Jules Berman

tags: Perl programming for medicine and biology, nomenclature, thesaurus, nlm, medical subject headings, open source, medical indexing, medical data retrieval, medical informatics, biomedical informatics, national library of medicine
Science is not a collection of facts. Science is what facts teach us; what we can learn about our universe, and ourselves, by deductive thinking. From observations of the night sky, made without the aid of telescopes, we can deduce that the universe is expanding, that the universe is not infinitely old, and why black holes exist. Without resorting to experimentation or mathematical analysis, we can deduce that gravity is a curvature in space-time, that the particles that compose light have no mass, that there is a theoretical limit to the number of different elements in the universe, and that the earth is billions of years old. Likewise, simple observations on animals tell us much about the migration of continents, the evolutionary relationships among classes of animals, why the nuclei of cells contain our genetic material, why certain animals are long-lived, why the gestation period of humans is 9 months, and why some diseases are rare and other diseases are common. In “Armchair Science”, the reader is confronted with 129 scientific mysteries, in cosmology, particle physics, chemistry, biology, and medicine. Beginning with simple observations, step-by-step analyses guide the reader toward solutions that are sometimes startling, and always entertaining. “Armchair Science” is written for general readers who are curious about science, and who want to sharpen their deductive skills.

Friday, February 22, 2008

Ruby, Perl, and Python medical autocoding

The other day, I created a very large web page that included 20,000 PubMed abstracts and the autocoded output for each.

The web page was apparently too large for some people to view, so I cut it down to show about 10,000 autocoded samples, along with the code for the autocoder in Ruby, Perl and Python. It is all available at:

http://www.julesberman.info/rubycode.htm


For anyone unfamiliar with medical autocoding, a medical autocoder is a software program capable of parsing large collections of medical records (e.g. radiology reports, surgical pathology reports, autopsy reports, admission notes, discharge notes, operating room notes, medical administrative emails, memoranda, manuscripts, etc.) and capturing the medical concepts contained in the text.

The term "autocoding" should be distinguished from "computer-assisted manual coding." Health care workers may use a software enhancement of their Hospital Information Systems to code a section of text as they enter reports into the computer system. Typically, candidate terms and term codes [from a medical nomenclature] are displayed on the same screen as the entered report. The person entering text is often given the option of editing the proffered codes. This process should not be confused with "autocoding" and is not equivalent to the fully automatic and large-scale coding required by biomedical informaticians.

Finding all the concepts in a corpus of text is a necessary and early step in all data mining efforts. The autocoded terms can be used individually as index terms for the document, on a record-by-record basis to produce a concept "signature" that is highly specific for each report, or collectively to relate the frequency of terms within records with the frequency of terms in the aggregate document.

The simple autocoder provided (in Perl, Python, and Ruby programming languages) is fast (about 100 kilobytes of text per second) and nearly perfect. You can check the output yourself for accuracy (in neoplasm terms extracted and coded). A minor modification of the scripts will accommodate any nomenclature for which terms are assigned concept-code numbers.

- Jules Berman

key words: medical software, nomenclature, medical datamining, perl programming, ruby programming, python programming, biomedical informatics, medical informatics, autcoding, autocoder, medical autocoding
Science is not a collection of facts. Science is what facts teach us; what we can learn about our universe, and ourselves, by deductive thinking. From observations of the night sky, made without the aid of telescopes, we can deduce that the universe is expanding, that the universe is not infinitely old, and why black holes exist. Without resorting to experimentation or mathematical analysis, we can deduce that gravity is a curvature in space-time, that the particles that compose light have no mass, that there is a theoretical limit to the number of different elements in the universe, and that the earth is billions of years old. Likewise, simple observations on animals tell us much about the migration of continents, the evolutionary relationships among classes of animals, why the nuclei of cells contain our genetic material, why certain animals are long-lived, why the gestation period of humans is 9 months, and why some diseases are rare and other diseases are common. In “Armchair Science”, the reader is confronted with 129 scientific mysteries, in cosmology, particle physics, chemistry, biology, and medicine. Beginning with simple observations, step-by-step analyses guide the reader toward solutions that are sometimes startling, and always entertaining. “Armchair Science” is written for general readers who are curious about science, and who want to sharpen their deductive skills.

Wednesday, February 13, 2008

Ruby, Perl and Python medical autocoders

In the past two days on this blog, I've provided very short, fast, and accurate medical autocoders in Ruby and Perl. I thought I might as well offer the equivalent Python script. The Python script runs about twice as fast as either the Ruby or the Perl script.

The Ruby, Perl and Python scripts and their equivalent output are provided at:

http://www.julesberman.info/coded.htm

They are distributed under a GNU license.

All three scripts use a public domain file of 20,000 PubMed Citations, available at:

http://www.julesberman.info/tumorabs.txt

They all use an external tumor nomenclature contained within the Neoplasm Classification and available as a gzipped XML file distributed under a GNU license at:

http://www.julesberman.info/neoclxml.gz

- Jules Berman
Science is not a collection of facts. Science is what facts teach us; what we can learn about our universe, and ourselves, by deductive thinking. From observations of the night sky, made without the aid of telescopes, we can deduce that the universe is expanding, that the universe is not infinitely old, and why black holes exist. Without resorting to experimentation or mathematical analysis, we can deduce that gravity is a curvature in space-time, that the particles that compose light have no mass, that there is a theoretical limit to the number of different elements in the universe, and that the earth is billions of years old. Likewise, simple observations on animals tell us much about the migration of continents, the evolutionary relationships among classes of animals, why the nuclei of cells contain our genetic material, why certain animals are long-lived, why the gestation period of humans is 9 months, and why some diseases are rare and other diseases are common. In “Armchair Science”, the reader is confronted with 129 scientific mysteries, in cosmology, particle physics, chemistry, biology, and medicine. Beginning with simple observations, step-by-step analyses guide the reader toward solutions that are sometimes startling, and always entertaining. “Armchair Science” is written for general readers who are curious about science, and who want to sharpen their deductive skills.

Thursday, January 31, 2008

More on "iform" words in UMLS

Here is the Perl script that extracted the "iform" words from UMLS:


#!/usr/local/bin/perl
open (TEXT, "MRCONSO");
$line = " ";
while ($line ne "")
{
$line = <TEXT>;
next if ($line !~ /ENG/);
if ($line =~ /\b[a-z]+iform\b/i)
{
$term = lc($&);
$subhash{$term}++;
}
}
foreach $key (sort keys %subhash)
{
print "$key\n";
}
exit;


The MRCONSO file (previously called the Mr. Con file) is the large (greater than 800 Megabyte) UMLS Metathesaurus file that contains all of the metathesaurus terms. It is available free from the U.S. National Library of Medicine, but you need to register and complete an online license agreement before they will release the metathesaurus files to you.

The Perl script (above) can be easily modified for simple extraction projects. If you're interested in learning Perl to help you with biomedical projects, you might want to read my book, Methods in Medical Informatics: Fundamentals of Healthcare Programming in Perl, Python, and Ruby (Chapman & Hall/CRC Mathematical and Computational Biology).

Here is the complete list of "iform" words:

acneiform
acniform
ansiform
apoplectiform
arciform
bacilliform
canaliform
cerebriform
chancriform
choreiform
chyliform
coliform
coralliform
cordiform
cribiform
cribriform
cruciform
cuciform
cuneiform
cupuliform
curariform
dendriform
dermiform
disciform
emboliform
epileptiform
equiform
falciform
fetiform
filariform
filiform
filliform
flagelliform
framboesiform
fundiform
fungiform
fusiform
gadiform
gelatiniform
gigantiform
gyriform
herpetiform
hydatidiform
ichthyosiform
intercuneiform
juxtarestiform
kaposiform
lentiform
licheniform
moniliform
morbilliform
multiform
myrtiform
naviculocuneiform
neuralgiform
nonhydatidiform
noviform
pampiniform
pectiniform
perciform
piriform
pisiform
pityriasiform
plexiform
prepiriform
prepyriform
proteiform
psoriasiform
punctiform
pyriform
rediform
reniform
restiform
retiform
retrolentiform
rhabditiform
rubelliform
sacciform
scarlatiniform
scarletiniform
schizophreniform
sclerodermiform
scorpaeniform
spongiform
storiform
subcuneiform
sublentiform
unciform
uniciform
uniform
valpiform
varicelliform
varioliform
vermiform
verruciform
vitelliform
zosteriform

In June, 2014, my book, entitled Rare Diseases and Orphan Drugs: Keys to Understanding and Treating the Common Diseases was published by Elsevier. The book builds the argument that our best chance of curing the common diseases will come from studying and curing the rare diseases.



I urge you to read more about my book. There's a generous preview of the book at the Google Books site. If you like the book, please request your librarian to purchase a copy of this book for your library or reading room.

- Jules J. Berman, Ph.D., M.D. tags: common disease, orphan disease, orphan drugs, rare disease, disease genetics, biomedical informatics, perl programming

Friday, January 4, 2008

Google preview of Perl Programming for Medicine and Biology

"Google Books" has chosen my work, Perl Programming for Medicine and Biology (2007), for limited preview.

You can browse the table of contents and read selected excerpts from the book. This comes about as close as in in-store browse as anyone might want, and I'm very grateful that Google provides this service. I have no idea how they choose which books get previewed, but I wish they would do it for every book-in-print.

-Jules Berman