v1.17 Release
METL now understands what your data is — not just what your file says it is. We also took our first step towards building an extractor template for you automatically. v1.17 also brought Transparency in Coverage data into the database and loaded our first network's TiC data.
New Features and Improvements
Automatic template building
Automatic template building takes its next big step with v1.17.
There are 2 capabilities to support this:
Data type analysis. The new dataType package reads the actual values in every column and classifies them into one of 30+ kinds — dates, NPIs, HCPCS and revenue codes, diagnosis codes, DRGs, NDCs, tax IDs, ZIPs, cities, first and last names, organization names, phone numbers, email addresses, addresses, and more. It doesn't rely on the column header, because trading partner headers are frequently wrong, cryptic, or missing entirely. It reads the data.
Name and city detection is backed by word lists we compiled and generate into the package — roughly 150,000 given names, 162,000 surnames, and 14,000 city words. That's what lets METL tell a LAST_NAME column from an ORG_NAME column when both contain nothing but text.
Column matching. Once every column has a type, METL scores each one against every field the wizard needs to fill and picks the best assignment. The matcher weighs the detected data type, the expected header words and patterns, and handles conflicts between columns when they appear to match the same input column. We've worked hard to ensure that a file with both a billing provider NPI and a rendering provider NPI won't get them backwards.
The result is a candidate list per field, with METL's pick pre-selected. In the next release you'll be able to use the UI to pick from the list of matches instead of building from scratch.
Supporting this, we added a header pattern type to the schema, so templates can describe how to find their headers with contains, whole, start, and end anchors rather than assuming row one.
User-definable value mapping
Trading partners describe the same thing a dozen different ways. M/F, Male/Female, 1/2 — all gender. Previously, translating those into our enums meant a code change.
v1.17 adds a new package to support user-definable value mapping. Currently, we use custom SQL scripts in csvloader to do value mapping like this. Coming in the next release, we'll update csvloader to support custom value mapping from the UI. That will allow you to define, per load, how source values map to any of 15 target enum types, including claim type, gender/sex, relationship, coverage level, marital status, employment status, insurance line, benefit status, prior authorization, unit type, and more. The mappings are validated in the database, so a mapping that names a value the enum doesn't have is rejected at write time rather than at load time.
The design distinguishes a source value mapped to NULL from a source value that matched nothing at all, which matters when you're auditing why a field came out empty.
Transparency in Coverage — and our first carrier data load with it
In v1.16 we shipped MRF table-of-contents parsing. In v1.17 we built the rest.
The database now has a full home for Transparency in Coverage data: transparency files, plans, in-network items, negotiated rates and prices, provider groups, provider references, and NPI sets, along with the supporting enum types for billing class, negotiation arrangement, negotiated type, plan market, plan ID type, and TIN type.
Alongside the schema, we added a new anthem downloader tool. It can download and load Anthem's published machine-readable files. These files run to tens of gigabytes, so the loader handles concurrent downloads, concurrent database loading, resumable attempts, progress reporting, and filtering down to just the plan sponsors you care about.
Extractor UI
The Extractor UI kept moving fast this release:
- A new left navigation bar, which makes the wizard feel like a real application instead of a single-page form
- Data type analysis is now surfaced in the UI, so you can see what METL detected for each column
- Support for CSV column mapping objects, so the per-column
optionalanduseFirstMatchsettings we added in v1.14 are now editable from the UI instead of only from YAML inputs/getnow returns all templates regardless of whether they match the uploaded file, so you can start from a similar partner's template rather than from nothing- Fixes to the data type analysis request
PHI access auditing
We extended the audit work from v1.16 to cover uploaded file contents. Every read of an uploaded file's data is now logged with the access type — Read for raw data, Analyze for data type inference — along with which columns were touched. Data type analysis reads real PHI to do its job, so it gets logged like any other access.
Exporter
- Added the ability to add a file header and footer for exported files
Member matching
Group ID is now displayed in the member matching UI, on both the canonical member and candidate member tables. We also fixed a StaleReactionError that could occur on the /matching route.
Testing and Deployment
Automatic subrepo sync
Subrepos now sync automatically instead of drifting until someone ran it manually.
Go lints are clean
We fixed the last of our outstanding Go lint violations. This allowed us to remove betterer from the pipeline for Go lints. The Go side of the codebase now lints without any issues.
Housekeeping
- A fix to
pkg/schedule - Fixed a
group_loadcase inconsistency which caused plans in the plan_match file with lowercase characters to never match - Updated Carbon Svelte packages and a large batch of npm and Go dependencies