Friday, February 29, 2008

The linguistic basis for the doublet medical autocoder and scrubber (Part 1)

In yesterday's blog, I presented a very fast (0.2 Megabytes per second) combined medical autocoder and medical scrubber. The software application is written in a few dozen lines of Perl. The output of the autocoder/de-identification software on a corpus of 95,260 PubMed Citations is available.

In the next few days, I'm going to try to explain the linguistic basis of how this software works.

First, you need to understand that medical terms occur primarily as phrases, not as single words.

The Neoplasm Classification contains over 135,000 names of neoplasms. Of these 135,000 terms, only about 500 are single-word terms.

Here is the list of about 500 single word terms in the Neoplasm Classification. The Neoplasm Classification is available in its entirety as a gzipped xml file.


List of single word terms from the Neoplasm Classification:

"acanthoma, achrochordon, acml, acrochordon, adamantinoma, adenoacanthoma, adenocarcinofibroma, adenocarcinoma, adenofibroma, adenolipoma, adenolymphoma, adenoma, adenomyoepithelioma, adenomyoma, adenopathy, adenosarcoma, aesthesioneuroblastoma, aesthesioneurocytoma, aesthesioneuroepithelioma, aild, allp, ameloblastoma, amml, androblastoma, angioblastoma, angioendothelioma, angiofibroma, angioglioma, angiokeratoma, angioleiomyoma, angiolipoma, angioma, angiomyofibroblastoma, angiomyolipoma, angiomyoliposarcoma, angiomyoma, angiomyosarcoma, angiomyxolipoma, angiomyxoma, angiosarcoma, anll, apl, apml, apudoma, argentaffinoma, arrhenoblastoma, asthesioneuroblastoma, asthesioneurocytoma, astroblastoma, astrocytoma, astroglioma, asymptomatic, atll, atypia, baltoma, basalioma, basiloma, blastoma, bnct, branchioma, calc, calcification, cancer, carcinofibroma, carcinoid, carcinoma, carcinosarcoma, cavernoma, cementoblastoma, cementoma, ceruminoma, chemodectoma, chloroleukaemia, chloroleukemia, chloroma, chloromyeloma, chlorosarcoma, cholangioadenoma, cholangiocarcinoma, cholangiohepatoma, cholangioma, choledochocele, choledochocoele, choledochocyst, cholesteatoma, chondroblastoma, chondroma, chondrosarcoma, chondrosteoma, chorangioma, chordocarcinoma, chordoepithelioma, chordoma, chorioadenoma, chorioangioma, choriocarcinoma, chorioepithelioma, chorionepithelioma, chromaffinoma, cml, collagenoma, comedocarcinoma, corticotropinoma, cpnet, craniopharyngioma, cylindroma, cyst, cystadenocarcinoma, cystadenofibroma, cystadenoma, cystoma, dentinoma, deposit, dermatofibroma, dermatofibrosarcoma, dermatomyofibroma, dermoid, desmoid, diktyoma, dipnech, dnet, dyscrasia, dysembryoma, dysgerminoma, dysplasia, ecchondroma, ectomesenchymoma, elastofibrolipoma, elastofibroma, elst, embryoma, enchondroma, endometrioma, endometrium, endotheliosarcoma, enteroglucagonoma, ependymoblastoma, ependymoma, epithelioma, erythraemia, erythremia, erythrocythaemia, erythrocythemia, erythrocytophagy, erythrodysplasia, erythroleukaemia, erythroleukemia, erythrophagia, erythroplakia, esthesioneuroblastoma, esthesioneurocytoma, esthesioneuroepithelioma, fibroadenoma, fibroepithelioma, fibrofolliculoma, fibroid, fibroleiomyoma, fibrolipoma, fibroliposarcoma, fibroma, fibromyoma, fibromyxolipoma, fibromyxoma, fibromyxosarcoma, fibroodontoma, fibrosarcoma, fibrothecoma, fibroxanthogranuloma, fibroxanthoma, fibroxanthosarcoma, gammopathy, gammapathy, gangliocytoma, ganglioglioma, ganglion, ganglioneuroblastoma, ganglioneurofibroma, ganglioneuroma, gangliorhabdomyosarcoma, gant, gastrinoma, gemistocytoma, germinoblastoma, germinoma, gist, glioblastoma, gliofibroma, glioma, glioneuroma, gliosarcoma, glomangioma, glomangiomyoma, glomangiopericytoma, glomangiosarcoma, glucagonoma, gonadoblastoma, gonadotrophinoma. gonocytoma, granulocytopenia, gtni, gynandroblastoma, haemangioblastoma, haemangioendothelioma, haemangioma, haemangiopericytoma, haemangiosarcoma, haemolymphangioma, hamartoma, hemangioblastoma, hemangioendothelioma, hemangioma, hemangiopericytoma, hemangiosarcoma, hemolymphangioma, hepatoblastoma, hepatocarcinoma, hepatocholangiocarcinoma, hepatoma, hgsil, hibernoma, hidradenocarcinoma, hidradenoma, hidroacanthoma, hidrocystoma, histiocytoma, hlrcc, hsil, hygroma, hypernephroma, hyperplasia, hyperplastic, hyperprolactinaemia, hyperprolactinemia, hypersensitivity, hyperthyroidism, idl, igcnu, immunoblastoma, immunocytoma, immunodeficiency, infertility, insulinoma, insuloma, ipmt, itgcn, jgct, keloid, keratoacanthoma, keratoameloblastoma, leiomyoblastoma, leiomyofibroma, leiomyoma, leiomyomata, leiomyosarcoma, lentigo, leptomeningioma, leucaemia, leucoplakia, leucorrhoea, leukaemia, leukemia, leukoplakia, leukorrhagia, leukorrhea, leukorrhoea, lipoadenoma, lipoblastoma, lipogranuloma, lipoleiomyoma, lipoma, lipomata, liposarcoma, lphd, luteinoma, luteoma, lymphadenoma, lymphadenopathy, lymphangioendothelioma, lymphangioleiomyoma, lymphangioma, lymphangiomyoma, lymphangiosarcoma, lymphepithelioma, lymphoblastic, lymphoblastoma, lymphocytoma, lymphoepithelioma, lymphoma, lymphosarcoma, macroglobulinaemia, macroglobulinemia, macroprolactinoma, maffucci, malignancy, malnutrition, malt, maltoma, mantleoma, masculinovoblastoma, mastocytoma, medulloblastoma, medullocytoma, medulloepithelioma, medullomyoblastoma, melanoacanthoma, melanoameloblastoma, melanocarcinoma, melanocytoma, melanoma, melanomeloblastoma, melanosarcoma, meningioma, mesenchymoma, mesonephroma, mesothelioma, metaplasia, mgct, micca, microglioma, microinvasive, microprolactinoma, milia, mmgct, mnti, mole, mucocele, myeloblastoma, myelodysplasia, myelolipoma, myeloliposarcoma, myeloma, myelosarcoma, myelosuppression, myoblastoma, myoepithelioma, myofibroblastoma, myofibroma, myofibrosarcoma, myolipoma, myoma, myopericytoma, myosarcoma, myxofibroma, myxofibrosarcoma, myxolipoma, myxoliposarcoma, myxoma, myxosarcoma, naevi, naevoxanthoendothelioma, ncmh, neoplasia, neoplasm, nephroblastoma, nephroma, nephropathy, nesidioblastoma, neurilemmoma, neurilemmosarcoma, neurilemoma, neurinoma, neuroastrocytoma, neuroblastoma, neurocytoma, neuroepithelioma, neurofibroma, neurofibrosarcoma, neurolipocytoma, neuroma, neuronaevi, neuronevi, neurosarcoma, neurotensinoma, neurotheceoma, neurothecoma, neurothekeoma, neutropenia, nevi, nevocarcinoma, nevoxanthoendothelioma, nlphd, odontoameloblastoma, odontoma, oligoastrocytoma, oligodendroblastoma, oligodendroglioma, ollier, oncocytoma, orchioblastoma, oscst, osteoblastoma, osteochondroma, osteochondromyxoma, osteochondroplastica, osteochondrosarcoma, osteoclastoma, osteodermia, osteofibrosarcoma, osteoma, osteosarcoma, pachydermatocele, pancreatoblastoma, pancytopenia, panin, papilloma, parachordoma, paraganglioma, pecoma, pericytoma, perifolliculoma, perineurioma, perithelioma, phaeochromoblastoma, phaeochromocytoma, pheochromoblastoma, pheochromocytoma, piloleiomyoma, pilomatricoma, pilomatrixoma, pinealblastoma, pinealoblastoma, pinealocytoma, pinealoma, pineoblastoma, pineocytoma, pituicytoma, plasmacytoma, pnet, pneumoblastoma, pneumocytoma, polycythaemia, polycythemia, polyembryoma, polyp, porocarcinoma, poroma, ppnet, precancer, preleukaemia, preleukemia, prelymphoma, premalignancy, preneoplasia, presarcoma, prolactinoma, psammoma, pseudolymphoma, pseudoneuroma, pstt, ptgc, ptyalocele, raeb, raem, ranula, reah, reninoma, reticulohistiocytoma, reticulolymphosarcoma, reticulosarcoma, retinoblastoma, retinocytoma, retinoma, rhabdomyoma, rhabdomyosarcoma, sarcoma, schimmelbusch, schwannoma, sebaceoma, sega, seminoma, settle, sgat, siadh, sialoblastoma, sialocele, sinonasalhaemangiopericytoma, sinonasalhemangiopericytoma, somatostatinoma, somatotropinoma, spermatocytoma, spiradenocarcinoma, spiradenoma, splenomegaly, spongioblastoma, spongioneuroblastoma, sterile, sterility, stromomyoma, sturge, subependymoma, sympathicoblastoma, sympathicogonioma, sympathogonioma, symptom, syncytioma, synovioma, syringadenoma, syringoadenoma, syringocystadenoma, syringocystoma, syringofibroadenoma, syringoma, teratocarcinoma, teratocarcinosarcoma, teratoma, thecoma, thrombocythaemia, thrombocythemia, thymolipoma, thymoma, trauma, trichilemmocarcinoma, trichilemmoma, trichoadenoma, trichoblastoma, trichodiscoma, trichoepithelioma, trichofolliculoma, trichogerminoma, tricholemmoma, tumor, tumour, undifferentiated, vain, verruca, verrucae, vipoma, wegener, werner, xanthelasma, xanthofibroma, xanthogranuloma, xanthoma, xanthosarcoma"

The consequences of a terminology that consists almost exclusively as multi-word phrases will impact the design of software that extracts medical terms from large text files. (to be continued).

- Jules Berman

key words: deidentification, medical nomenclature, term extraction, scrubber, scrubbing
Science is not a collection of facts. Science is what facts teach us; what we can learn about our universe, and ourselves, by deductive thinking. From observations of the night sky, made without the aid of telescopes, we can deduce that the universe is expanding, that the universe is not infinitely old, and why black holes exist. Without resorting to experimentation or mathematical analysis, we can deduce that gravity is a curvature in space-time, that the particles that compose light have no mass, that there is a theoretical limit to the number of different elements in the universe, and that the earth is billions of years old. Likewise, simple observations on animals tell us much about the migration of continents, the evolutionary relationships among classes of animals, why the nuclei of cells contain our genetic material, why certain animals are long-lived, why the gestation period of humans is 9 months, and why some diseases are rare and other diseases are common. In “Armchair Science”, the reader is confronted with 129 scientific mysteries, in cosmology, particle physics, chemistry, biology, and medicine. Beginning with simple observations, step-by-step analyses guide the reader toward solutions that are sometimes startling, and always entertaining. “Armchair Science” is written for general readers who are curious about science, and who want to sharpen their deductive skills.