About 80% of medical data remains unstructured and largely untapped after its creation, such as text, images, and signals. Since handling this type of data is challenging for Electronic Medical Records and most hospital information systems, it is often overlooked, unsaved, or abandoned in medical centers. Despite the vast amounts of data generated in hospitals, it remains disconnected from medical big data research and the healthcare AI industry. Therefore, before advancing AI in healthcare, especially machine learning-based technologies, it is essential to first address the management of this unstructured data in healthcare systems.
Data purging is necessary to eliminate data that surpasses the retention period defined in the persistence settings. This process is managed through a database job created during the application’s installation. Data is purged when the current date exceeds the report date plus the specified persistence duration.
In essence, data purging involves securely and permanently removing data from a system, such as a database or storage device. Purging outdated or irrelevant data helps reduce storage costs and minimizes the risk of data breaches. Additionally, organizations may need to purge data to comply with regulations like the General Data Protection Regulation (GDPR).
A few industry-leading purge features built into Fovea, include:
- Purge Rules
- Manual Purge
- Proof of Purge
- Purge Delay
Organizations often create data retention policies that outline how long specific data should be kept before it is deleted. These policies must balance business requirements with relevant legal obligations.
In line with these policies, organizations must periodically identify and remove data that is no longer necessary. This purgeable data may include information that is no longer useful for business purposes, sensitive data that should not be stored for extended periods, or data that must be erased to meet legal requirements.
Difference Between Data Purging and Data Archiving
While data purging and data archiving both deal with managing outdated data, they serve different purposes:
- Data Purging: Permanently deletes data that is no longer needed.
- Data Archiving: Moves data to long-term storage for future reference or compliance purposes.
Common methods for data purging include securely deleting data by overwriting it multiple times with random bits, making it nearly impossible for recovery tools to retrieve the information.
For physical storage devices like hard drives or SSDs, purging may involve physically destroying the media through shredding or incineration.
Alternatively, data may be marked as deleted in the file system, making it inaccessible through standard methods. However, until it is overwritten by new data, it remains potentially recoverable.
Why is Data Purging Important?
Data purging is crucial for several reasons:
- Enhanced System Performance Accumulated data can slow down system performance over time. By removing unnecessary data, system efficiency is restored, leading to faster processing and smoother operations.
- Cost Savings Storing excessive data can incur significant costs. Purging outdated or irrelevant data frees up valuable storage, helping organizations lower expenses, especially those with limited storage capacity.
- Regulatory Compliance Industries often require adherence to strict data retention rules. Data purging supports compliance by eliminating outdated data, reducing the risk of violations and legal complications.
- Streamlined Data Management Handling vast amounts of data can be complex. Data purging simplifies this process by keeping databases organized and easy to navigate, facilitating better access to relevant information.
How is Data Purging Performed?
The data purging process involves several key steps to ensure that outdated data is properly identified, removed, and verified. Here’s a breakdown of how data purging is carried out:
Identifying Data to Purge: Determine which data should be removed based on factors like age, relevance, or other specified criteria.
Backing Up Critical Data: Ensure that important data is backed up before purging to avoid accidental loss.
Deleting Unnecessary Data: Use tools like SQL commands or scripts to delete the identified data.
Verifying the Purge: Confirm that the data has been successfully removed and that no critical data has been deleted by mistake.
Data Purging in ETL
In Extract, Transform, Load (ETL) processes, data purging refers to the removal of data that is no longer required after it has been processed and loaded into the target system. This practice helps ensure that only relevant data is retained, enhancing the efficiency of ETL operations.
Examples of Effective Data Purging:
Company A improved system efficiency by purging outdated customer data, which led to faster processing and lower storage costs.
Company B complied with data retention policies by regularly purging obsolete financial records, mitigating the risk of non-compliance.
Case Studies on the Benefits of Data Purging:
Case Study 1: A healthcare organization used a data purging strategy to remove old patient records, enhancing data management and ensuring compliance with regulations.
Case Study 2: An e-commerce company deleted outdated transaction data, optimizing storage and boosting system performance.
- Getting Started with Secure Document Destruction Services
- To ensure compliance with document disposal regulations, it’s essential to assign a dedicated individual at your facility to oversee the process. If you’re unsure where to begin, TriHaz Solutions offers a free consultation to guide you in establishing compliant procedures. During the consultation, we will also assist in determining the necessary staff training and how often it should be updated.
- Our secure document destruction services adhere to NAID best practices and include:
- Regular shredding services, which can be bundled with your existing regulated waste management services.
- Specialized shredding, including the destruction of x-rays and discarded pill bottles.
- On-demand or one-time shredding services to thoroughly purge boxes, file cabinets, and other storage containers.
- Secure disposal of PHI (Protected Health Information) documents.
- HIPAA/HITECH-compliant hardware and media destruction.
- Secure document storage solutions, such as cabinets and carts, to manage highly sensitive document collection and storage needs.
Purge Rules
Fovea is designed to accommodate a wide range of “if-then” scenarios based on record retention policies, offering flexibility in how purge rules are created, managed, and exceptions are handled. Criteria for record disposal can include factors such as the date of last service, discharge date, date of birth, or the type of document or data (e.g., behavioral health records). Exceptions to these rules may apply in cases like legal holds, RAC audits, clinical research, or other situations that require preserving medical records.
- Automatic or Manual Purge
Purging can be set up as either an automatic or manual process. In automatic mode, the system follows predefined purge rules, identifies records that meet the deletion criteria, and schedules them for purging at a designated time without user involvement. Manual purging, on the other hand, gives users control over when the process occurs and which rules are applied at that time.
- Proof of Purge
A certificate of destruction is provided to verify that a record once existed. The certificate details the reason for purging, along with the date and time of the event (e.g., the record reached its 10-year retention period on 12/31/16 at 10:00 am ET).
- Purge Delay and Un-Purging
When a purge routine is processed, a “soft purge” initially occurs, meaning the records are no longer visible in the archive but can be restored within a user-defined period (known as the purge delay). After this period expires, the records undergo a “hard purge,” permanently deleting them from the archive in accordance with National Institute of Standards and Technology (NIST) guidelines.
If your healthcare organization is decommissioning legacy EMR or ERP systems, patient and employee data should be securely extracted, migrated, and archived to meet your retention and destruction policies. To explore Health Data Archiver’s purging features, schedule a demo with one of our archiving experts.
- The Urge to Purge
While patient records can be kept indefinitely, this approach can be costly. A more strategic method is to calculate the rate at which patient records can be purged from archives to establish cost baselines and potential return-on-investment. Some organizations take a cautious stance and retain PHI forever, while others adhere to specific retention periods and purge the data once that time expires. Risk management teams generally prefer data to be purged as soon as legally allowable, as legacy data is seen as a potential liability.
Most healthcare data archiving solutions include features to ensure retired data is securely stored and retained according to applicable regulations and organizational policies. A policy engine can automatically enforce retention rules, purging data once its retention period has passed to mitigate legal risks. Legal hold features also ensure relevant data is preserved for legal proceedings, even after the retention period ends.
Challenge:
A key challenge with EHR purging is that most EHR vendors do not offer built-in archiving or purge features for EHR charts. EHR vendors could look to PACS and vendor-neutral archive providers, who have long dealt with large file sizes and developed efficient solutions for archiving and purging. The question remains whether EHR data should be purged from the main system and transferred to document management software. These are complex issues that don’t have straightforward answers.