REIDENTIFICATION UNDER HIPAA

Reidentification is a term casually applied to any instance whereby information can be linked to a specific person, after the links between the information and the person associated with the information have been removed. Used this way, the term reidentification connotes an insufficient deidentification process. In the health care industry, the term "reidentification" means something else entirely. In the U.S., regulations define "reidentification" under the "Standards for Privacy of Individually Identifiable Health Information.”(1) Therein, reidentification is a legally sanctioned process whereby deidentified records can be linked back to their human subjects, under circumstances deemed legitimate and compelling, by a privacy board. Reidentification is typically accomplished via the use of a confidential list of links between human subject names and deidentified records, held by a trusted party. In the healthcare realm, when a human subject is identified through fraud, trickery, or through the deliberate use of computational methods to break the confidentiality of insufficiently deidentified records (ie, hacking), the term "reidentification" would not apply.(2)

References

1. Department of Health and Human Services. 45 CFR (Code of Federal Regulations), parts 160 through 164. Standards for privacy of individually identifiable health information (final rule). Fed Regist 2000;65(250):82461–510.

2. Berman JJ. Principles of big data: preparing, sharing, and analyzing complex information. Morgan Kaufmann; 2013.

- Jules Berman (copyrighted material)

key words: identifier, deidentifier, identification, reidentification, privacy, HIPAA, confidentiality, medical records, jules j berman

Fast deidentifier that preserves punctuation

On Tuesday, Jan 17, 2008, I provided a very simple, fast, and almost perfect medical record de-identifier perl script . The script uses a public domain list of about 200,000 word doublets.

A public domain file shows the output this de-identifier with an input of about 15000 PubMed medical citations. PubMed citations are an excellent way to test de-identifiers because they are non copyrighted, they contain lots of medical vocabulary, and they are full of identiifiers (the names of the authors).

The provided output file does not preserve the punctuation in the original text.

It is easy to modify the Perl script to preserve case (lowercase, uppercase) and punctuation from the original text, and the output of the modified script is also available.

As with all my distributed scripts, the following disclaimer applies:

The perl script for deidentifying text using the doublet method is provided by its creator, Jules J. Berman, "as is", without warranty of any kind, expressed or implied, including but not limited to the warranties of merchantability, fitness for a particular purpose and noninfringement. in no event shall the author or copyright holder be liable for any claim, damages or other liability, whether in an action of contract, tort or otherwise, arising from, out of or in connection with the software or the use or other dealings in the software.

The revised Perl script is shown here:

#!/usr/local/bin/perl
$begin = time();
open(TEXT,"doublets.txt")||die"cannot";
$line = " ";
while ($line ne "")
  {
  $line = $getline = <TEXT>;
  $getline =~ s/\n//;
  $doublethash{$getline}= "";
  }
$end = time();
$totaltime = $end - $begin;
print STDERR "Time to create ";
print STDERR "the doublet hash is ";
print STDERR "$totaltime seconds.\n\n";
close TEXT;
$begin = time();
open(TEXT,"pathol5.txt")||die"cannot";
open(STDOUT,">pathol5.out")||die"cannot";
$line = " "; $oldthing = ""; $state = 0;
while ($line ne "")
   {
   $line = <TEXT>;
   next if ($line eq "\n");
   print "Original - $line" . "Scrubbed - " ;
   $line =~ s/\n$//;
   #$line =~ s/\n/ /o;
   my @linearray = split(/ +/,$line);
   push (@linearray, "lastword");
   foreach $thing (@linearray)
     {
     $originalthing = $thing;
     $thing = lc($thing);
     $thing =~ tr/a-z\'\-//cd;
     if ($oldthing eq "")
        {
        $oldthing = $thing;
        $originaloldthing = $originalthing;
        next;
        }
     $term = "$oldthing $thing";
     if (exists($doublethash{$term}))
        {
        print "$originaloldthing ";
        $oldthing = $thing;
        $originaloldthing = $originalthing;
        $state = 1;
        next;
        }
     if ($state == 1)
        {
        if ($thing eq "lastword")
          {
          print $originaloldthing;
          print "\n";
          $oldthing = "";
          $state = 0;
          next;
          }
        print "$originaloldthing ";
        $oldthing = $thing;
        $originaloldthing = $originalthing;
        $state = 0;
        next;
        }
     if ($state == 0)
        {
        if ($thing eq "lastword")
          {
          print "\*\.\n";
          $oldthing = "";
          next;
          }
        $punctuation = substr($originaloldthing,-1,1);
        if ($punctuation =~ /[a-zA-Z0-9]/)
           {
           $punctuation = "";
           }
        print "\*" . "$punctuation ";
        $oldthing = $thing;
        $originaloldthing = $originalthing;
        next;
        }
     }
   }
$end = time();
$totaltime = $end - $begin;
print STDERR "Time following ";
print STDERR "doublet hash creation";
print STDERR " is $totaltime seconds.";
exit;

- Jules Berman

---

My book, Principles of Big Data: Preparing, Sharing, and Analyzing Complex Information was published in 2013 by Morgan Kaufmann.

I urge you to explore my book. Google books has prepared a generous preview of the book contents.

tags: big data, metadata, data preparation, data analytics, data repurposing, datamining, data mining, deidentified, deidentifier, hipaa, medical de-identifier, medical scrubber, scrubbed text