Free Reference Data Scanner

Are you wrestling with reference data hiding in your data platform?

• Country codes in a Python dictionary.
• A cost-centre list pasted into three notebooks.
• A CASE statement mapping product codes to categories.
• An INSERT... VALUES(..) statement loading a metadata table.
• A spreadsheet someone uploads to a volume every month.

These are all examples of reference data, scattered across your data platform.

About

The Reference Data Scanner (refdata-scanner) is a free, open-source, read-only tool that finds reference data hiding in your data platform. It produces a detailed report in multiple formats, so you can assess how much ungoverned reference data exists on your data platform. It's a great way to identify and feeds governance and management strategies for your reference data assets.

What it finds

• CSV and Excel files in Catalog volumes, workspace folders (including Git folders), DBFS /FileStore, dbt seed folders, and files known only from lineage.
• Lookup-table candidates. Small, narrow tables with reference-data style names, or tables loaded from spreadsheets.
• Hardcoded reference data in code. Python dicts, lists and sets; spark.createDataFrame([...]) literals; .isin([...]) filters; F.when() chains; SQL CASE mappings, long IN (...) lists and VALUES tables — in notebooks, code files, saved queries and view definitions.
• Tables filled from literal rows. INSERT INTO ... VALUES, MERGE ... USING (VALUES ...) and CREATE TABLE AS VALUES.
• Duplicates and drift. The same list or mapping copied into several places, and copies that disagree — for example WA: West Australia vs Western Australia.

Three ways to run it

1. Five-minute SQL check. Import one SQL notebook into Databricks and Run all. It uses Unity Catalog system tables and the information schema to list spreadsheet-loaded tables, literal-row tables, hardcoded view mappings and lookup-table candidates. Nothing to install.
2. Full scan in a Databricks notebook (recommended). Import the zip, open 02_full_scan, choose a catalog, schemas and the workspace folders to scan, and run it. It uses your own identity, so there are no credentials to configure.
3. Command line. pip install the scanner and point it at a Databricks workspace or at local folders — a Git checkout of your notebooks, a dbt project or an exported workspace.

What you get

Each run writes an HTML and Markdown assessment report — headline numbers, key findings, duplicated and drifting sets, top candidates, and hotspots by location and owner — plus CSV inventories of every file, table and code finding.

Every candidate gets a score from 0 to 100 with the reasons spelled out. Mappings and inline tables score higher than plain lists, and reference-data style names, duplication, disagreeing copies, use in scheduled jobs and manual upload locations all add to the score. Things that only look like reference data, such as lists of column names or Spark configuration, are set aside rather than hidden.

Safe by design

• Read-only. It lists and reads; it never creates, changes or deletes anything on your platform.
• Metadata and code only. File contents are not read unless you turn that on, and even then only the header row and a row estimate are kept.
• Nothing leaves your environment. No telemetry. The only network calls are to your own platform's APIs, using the identity of the user running it.
• Open Source, small and readable. Plain Python you can read before you run it.

Supported today: Databricks and local folders. Snowflake and Microsoft Fabric are on the roadmap.

Found it all? Give it a home.

Most organizations that run the scanner find hundreds of lists, mappings and hierarchies with no owner, no versioning and no approval workflow. A dedicated Reference Data Management system like TitanRDM gives that data a governed system of record, with user access, versioning, approvals, data platform integration and a full audit history.