Data Pipelines – Encamp | Knowledge Base

What Are Data Pipelines?

Data pipelines are vital to Encamp's operations. They enable us to transform raw data sets from our customers into compliant Tier II reports. Whether the data is provided as an export from an internal Enterprise Resource Planning system, an Excel worksheet, or a third-party vendor, our Data Solutions team works with you to understand your data and automate its transformation into useful insights.

✏️

NOTE: The ultimate goal of an established data pipeline is the ability to seamlessly populate accurate and compliant chemical data on a Tier II report from the raw chemical inventory data collected throughout the year.

Process Overview

Our data pipeline process consists of the following steps:

Step 1: Data Collection

Encamp provides multiple methods for collecting your data. The two most common forms include direct upload within the Encamp application or cloud-based file sharing, such as Box.com.

Step 2: Data Cleaning

After your data is received, it is evaluated for gaps and cleaned to remove irrelevant data. This step ensures data accuracy by removing non-chemical products from inventory reports and correcting and removing corrupt, mislabeled, poorly formatted, duplicate, or incomplete data.

Step 3: Data Transformation

Your data is then transformed into a format suitable for ingestion into Encamp. This transformation process includes standardizing data, splitting or merging data, converting units of measurement into a standard format, and aggregating and consolidating data. This process ensures that the data is usable.

Step 4: Data Ingestion

Data ingestion pipelines further transform your data into predefined formats and deliver it to Encamp. This step automates some tasks that were previously performed manually, improving efficiency.

Step 5: Data Validation

Once your data is in Encamp, it is checked against various data and compliance rules to determine if the information falls within the acceptable range of values. This data validation is performed to ensure that the data collected fulfills jurisdictional requirements.

Quality Assurance and Control

The pipeline development process incorporates a variety of periodic checks to ensure that your data is accurate and consistent. These data quality checks are performed at each phase of pipeline development, ensuring that the final Tier II report is accurate and compliant.

Data Verification

Data verification steps are incorporated into each phase of the pipeline development process as a final measure. This stringent approach guarantees the accuracy and consistency of your data, providing confidence that your Tier II reports comply with all relevant standards and requirements.

Partnering with Our Customers

Our Data Solutions team works directly with you to clean and transform your raw data sets, ensuring data integrity and quality. As a customer, you will be assigned an Environmental Data Manager who will:

Manage the base product catalog

Manipulate raw data into an ingestible format

Incorporate your specific requirements and/or requests for data outputs

Communicate directly with you throughout the pipeline development process to include you in decisions about your environmental data

Conduct QA/QC and data verification at critical points throughout the development process