v1.16 Release

Since the beginning, the ETL in METL has always been about extract, transform and load. With v1.16, we're making ETL much more accessible. The Extractor UI is now read-write and we added an audit trail so you can see exactly who accessed what.

New Features and Improvements

The Extractor UI is now read-write

We've been promising this for a couple of releases and it's finally here. You can now create a new input template directly from the Extractor UI instead of hand-editing YAML.

The new workflow is:

1. Upload a file (or pick one that was previously uploaded — the UI now remembers).
2. METL analyzes the file and figures out the format and delimiters.
3. You review a preview of real data rows from your file.
4. You map the columns and save a brand new input template.
5. Run the database load and check the results right from the UI.
6. If there are issues with the results, delete it and re-run until you've got it right.

File analysis is one of the pieces we're most excited about over time. Eventually, we believe we'll be able to automate much of the process of mapping files. METL will make its best guess at what each column contains and you correct it from there. Our goal is to take new trading partner onboarding from months down to a few minutes. For now, we analyze the file to determine whether it is a delimited file or fixed-width and show a preview of the file.

The wizard steps are where the bulk of the work in the Extractor UI gets done today. The user is guided through the process of assigning columns in the input file to the METL database. Steps are fully configurable. This allows us to refine the steps to fill out a template to make it the easiest for users as we learn. Steps include:

• Bucket step - a big bucket where lots of related fields go together. Claims files can have hundreds of columns, so bucket steps allow us to group similar sorts of fields together. For example, we have a bucket step for all provider-related fields. We use these in a similar way as humans use piles when they are sorting a deck of cards. If you're sorting cards, you might sort by number, by suit, etc. Once you've done that sort, you'll probably sort again. This is how we use buckets for the UI.
• Map Output - This is where we assign an input column in the source file to where it belongs in the database. You might end up dragging a column named "Member Last Name" into the "claim_member.member.name_last" column in the database. Similar fields can be grouped together into tables to make it easy to find what you're looking for.
• Question - Sometimes we need to ask a question in order to know what to simplify what we show to the user in the Wizard. Question steps allow us to alter the UI based on the answer supplied
• Input form - Some things just require a person to type something in or pick something form a list of options. That's where an input form is helpful.

The Extractor UI also added a new data loads view. This allows you to load the data file that you've configured and see its status without leaving the wizard. If the results don't look like what you expected, you can delete the data from the database make the necessary fixes and run it again.

Job audit trail

With APIs comes the responsibility to track how the data is being used. We added a new job_data_access audit log that records one row per call to the data endpoint — who called it, when, and how many rows they received. A companion job_data_access_job table records exactly which jobs each call touched.

We also continued the jobs work started in v1.15:

• job_etl now tracks when a job was queued, not just when it started
• Reloads now prefer the succeeded job when there are multiple attempts, so a retried file no longer shows up as failed
• Extracted data is cleared when the extractor fails, so a partial extract can't be loaded by mistake

MRF parsing

We added support for parsing machine-readable file (MRF) tables of contents. Payers are required to publish their negotiated rates as MRFs under the Transparency in Coverage rule, and those files can be enormous. Parsing the table of contents is the first step toward pulling in the in-network rate files we actually care about.

Member matching

The member matching UI can now be filtered by solution, which makes it much easier to manage a queue of members.

Behind the scenes, patient matching got significantly faster and much lighter on memory. Large matching runs that previously pushed against memory limits now complete comfortably.

Exporter

We updated several exporter queries and fixed a deadlock that could occur when multiple exports ran at the same time.

Backend changes

• Snapshot sorting is dramatically faster
• We removed TRUNCATE TABLE statements that were forcing unnecessary table locks
• We switched to statement_timestamp() where transaction-level timestamps were giving misleading results
• Our test suite and build-dbs are both faster, thanks to better caching

View improvements

• member_eligibility_current_view — previously we took the single latest eligibility submission per trading partner. Several partners send multiple independent eligibility files (e.g. one per group ID), and taking only the newest file meant the other groups were silently dropped. The view now recognizes the file "version" in the filename and keeps the latest submission for each version.
• member_coverage_view — switched the plan sponsor joins from INNER JOIN to LEFT OUTER JOIN. Members without a plan sponsor match were being dropped from the view entirely.

Bug fixes

• Fixed an 834 parse error
• Fixed a Svelte LSP crash that was slowing down frontend development

Testing and Deployment

Atlas lints in Betterer

Our Atlas database lints now run through Betterer alongside the rest of our linting. That means schema issues are caught and tracked the same way as everything else in the codebase. Betterer helps us to continually reduce the number of lint violations over time instead of letting them sit. Database lints include things like ensuring each table has a foreign key, every column and every table has a comment, etc.

Dependency updates

We updated Carbon Components Svelte to the latest version.

METL DB Release on GitHub

METL-deploy on GitHub