DQOps

28/03/2024

Profiling CSV files in DQOps

This 60-second video shows the process of connecting a new data source, profiling the data, running data quality checks, and reviewing results.

Once the profiling is done, DQOps will keep monitoring the data quality of the CSV file and will alert you if any issues arise that make the file no longer valid.

Ready to transform your data quality? Experience DQOps for yourself!
Check out DQOps documentation: https://dqops.com/docs/

07/12/2023

Define data quality requirements from the data engineering perspective

The Data Engineering Team is responsible for managing the data warehouse or the data lake. The data quality issues of the incoming data may affect the stability of the data pipelines.

Well-designed data warehouses, ETL tools, and data pipelines report the progress of data pipelines and errors as logs. However, to fully understand the complexities of data processing, it is necessary to have a bird's eye view of the entire process.

This stage describes the key steps for gathering the data engineering requirements in the data quality area, introducing the data engineers to the data quality tool, and connecting the data quality tool to the data platform.

Learn more about the efficient data quality process in our Ebook (https://dqops.com/best-practices-for-effective-data-quality-improvement/).

09/10/2023

Fixing source data issues

As the first step, the Data Owner should check whether the problem is present in the source platform, such as an OLTP database, or if it is only in the target platform (data warehouse or data lake). If the source data is correct, the Data Owner contacts the Data Engineering Team to review the data pipelines.

However, if there is a problem with the source data, the Data Owner tries to solve it with the Data Producer, even involving external data suppliers or business users. If the problem cannot be resolved, the Data Owner may create a list of acceptable data quality exceptions.

Learn more about the efficient data quality process in our Ebook(https://dqops.com/best-practices-for-effective-data-quality-improvement/).

04/10/2023

Improving data quality KPIs

The Data Quality Team monitors data quality KPIs on newly created data quality dashboards. Any identified data quality issues must be reviewed with the Data Owner, who should take responsibility for the next steps.

The data quality issues present at the data source level can be fixed by the Data Producer. Issues caused by a bug in the data pipeline or an ETL process should be fixed by the Data Engineering Team. Once the problem is resolved, the Data Quality Team re-executes data quality checks.

If the issue cannot be fixed immediately, the Data Quality Team may adjust the alerting thresholds or acceptable levels of data quality KPIs.

Learn more about the efficient data quality process in our Ebook(https://dqops.com/best-practices-for-effective-data-quality-improvement/).

21/09/2023

Develop and deploy custom data quality checks.

A customizable data quality platform (such as DQO) should support the use of custom data quality check definitions, provided as custom SQL queries or implemented as a custom code.

Custom data quality checks should be reusable across tables. Custom
checks should not be hard coded for individual tables. To enable the reusability of custom data quality checks, The following example is a template of a data quality sensor that is counting rows with a non-negative column value.

Learn more about the efficient data quality process in our Ebook(https://dqops.com/best-practices-for-effective-data-quality-improvement/).

20/09/2023

Provide a list of KPIs to be monitored
Data quality sensors capture quality-related metrics from monitored data sources. These sensor readouts should be evaluated by data quality rules to detect outliers or measures that do not meet the required thresholds.

In DQO, the combination of the data quality sensor and data quality rule is called a data quality check. For long-term data quality monitoring, the data quality platform must measure the percentage of passed data quality checks within all executed data quality checks. This percentage of passed data quality checks is called a data quality KPI.

The expected result of calculating the data quality KPI at different grouping levels may look like the following tables:

Learn more about the efficient data quality process in our Ebook(https://dqops.com/best-practices-for-effective-data.../).

04/09/2023

Certain data quality requirements influenced by business users may require more complex data quality checks. These data quality checks should be separated and thoroughly analyzed. Below are the most common data quality requirements that should be satisfied by custom data quality checks.

᛫ Custom data formats. Column values must follow a complex pattern that is too complex to parse using regular expressions. These usually involve names that must follow a naming convention.

᛫ Multi-column checks. These data quality checks perform arithmetic operations across different columns. A simple example is a data quality check that verifies that a net_price + tax = total_price.

Learn more about the efficient data quality process in our Ebook(https://dqops.com/best-practices-for-effective-data-quality-improvement/).

Download DQO eBook to learn best practices for effective data quality improvement. Reach 100% data quality score.

29/08/2023

Assign the initial thresholds. The threshold represents just the expectations and beliefs about the current data quality status. The Data Owner or the Data Engineering Team may believe that there are no invalid rows, so the rule to count the number of invalid rows should be "equals 0".

The correct values will be validated in later steps. The thresholds should later be adjusted to a more reasonable value. The default alerting thresholds are raising data quality issues at the "error" severity level.

Learn more about the efficient data quality process in our Ebook(https://dqops.com/best-practices-for-effective-data-quality-improvement/).

Download DQO eBook to learn best practices for effective data quality improvement. Reach 100% data quality score.

12/12/2022

As data complexity increases, the importance of has never been greater. But getting started with can be challenging. Use our best practices to remove all data quality issues and reach a 100% data quality score.

Download our eBook to learn how to:
Set up a data quality monitoring process.
Define requirements for both a business and data engineering perspective.
Measure data quality across multiple dimensions, such as accuracy, validity, completeness, consistency, currency, or timeliness.
Detect and respond to data quality problems.

Download here: https://lnkd.in/dQBv_s2x 🔍

14/11/2022

We would like to invite you to the next meetup organised by Dqo.ai!

👉Does your data meet all required data quality dimensions?
👉Would you like to safely analyze the data quality on different clouds?
👉Do you spend too much time on managing data quality?

If you are interested in the answers to these questions, this meetup is sure to meet your expectations!🔍

We are happy to invite you to our meetup Data Quality process: How to meet data quality dimensions?🔍 We will have two hosts: 👉Piotr Czarnas - Experienced manager and sof