Verify the configuration files

Oct 13, 2025 by pinkandtrash.com 80 views

Introduction

In this article, we will explore the issue of unable to create lineage from a SQL query using the DataHub CLI and Python SDK. We will delve into the error messages, configuration files, and code snippets to identify the root cause of the problem and provide solutions to overcome it.

Error Message

When running the DataHub CLI command datahub ingest -c lineage_datahub.yaml, we encounter the following error message:

'Unable to emit metadata to DataHub GMS: Failed to validate record with class com.linkedin.dataset.UpstreamLineage:'

This error message indicates that there is a problem with validating the record, specifically with the UpstreamLineage class.

Configuration Files

Let's take a closer look at the configuration files used in the DataHub CLI and Python SDK.

YAML File (lineage_datahub.yaml)

source:
  type: sql-queries
  config:
    platform: "athena"
    query_file: "./queries.json"

sink: 
  type: "datahub-rest"
  config:
    server: "http://localhost:8080"

In this YAML file, we specify the source type as sql-queries and the platform as athena. We also point to the queries.json file for the query.

JSON File (queries.json)

{"query": "CREATE TABLE \"dev-datalake-gold-entities\".countries AS select * FROM\n \"dev-datalake-silver\".countries_hub_vw"}

In this JSON file, we have a single query that creates a table in the dev-datalake-gold-entities schema.

Python SDK Code

Let's examine the Python SDK code that attempts to obtain the lineage results.

from datahub.emitter.rest_emitter import DatahubRestEmitter
from datahub.ingestion.graph.client import DatahubClientConfig, DataHubGraph

query = """        
        INSERT INTO "dev-datalake-gold-entities".countries
        SELECT
        countries_hub.country_code,
        FROM
        "dev-datalake-silver".countries_hub_vw countries_hub
"""

gms_endpoint = "http://localhost:8080"
token = ""
graph = DataHubGraph(
    DatahubClientConfig(
        server=gms_endpoint,
        token=token,
    )
)

lineage = graph.parse_sql_lineage(sql=query, platform="athena")
print(lineage)


emitter = DatahubRestEmitter(
    gms_server=gms_endpoint,
    token=token,
)

# TODO : how to emit above lineage to datahub?

In this code, we create a DataHubGraph object and use the parse_sql_lineage method to obtain the lineage results. However, we are unsure how to emit this metadata to DataHub.

Solutions

Based on the error message and code snippets, we can try the following solutions:

Check the query syntax: Ensure that the query syntax is correct and follows the DataHub SQL syntax rules.
Verify the configuration files: Double-check the configuration files, including the YAML and JSON files, to ensure that they are correctly formatted and contain the necessary information.
Use the correct platform: Make sure that the platform specified in the configuration file matches the platform used in the query.
Emit metadata using the Python SDK: Use the DatahubRestEmitter class to emit the metadata to DataHub. We can modify the code to emit the lineage results using the emit method.

Example Code: Emitting Lineage Results using the Python SDK

from datahub.emitter.rest_emitter import DatahubRestEmitter
from datahub.ingestion.graph.client import DatahubClientConfig, DataHubGraph

query = """        
        INSERT INTO "dev-datalake-gold-entities".countries
        SELECT
        countries_hub.country_code,
        FROM
        "dev-datalake-silver".countries_hub_vw countries_hub
"""

gms_endpoint = "http://localhost:8080"
token = ""
graph = DataHubGraph(
    DatahubClientConfig(
        server=gms_endpoint,
        token=token,
    )
)

lineage = graph.parse_sql_lineage(sql=query, platform="athena")
print(lineage)

emitter = DatahubRestEmitter(
    gms_server=gms_endpoint,
    token=token,
)

# Emit the lineage results
emitter.emit(lineage)

In this example code, we use the DatahubRestEmitter class to emit the lineage results to DataHub.

Conclusion

In this article, we explored the issue of unable to create lineage from a SQL query using the DataHub CLI and Python SDK. We identified the root cause of the problem and provided solutions to overcome it. By checking the query syntax, verifying the configuration files, using the correct platform, and emitting metadata using the Python SDK, we can successfully create lineage from a SQL query in DataHub.

Additional Resources

DataHub Documentation: SQL Queries
DataHub Documentation: Lineage
DataHub GitHub Repository: acryl-datahub
DataHub Lineage: Frequently Asked Questions =====================================================

Q: What is DataHub Lineage?

A: DataHub Lineage is a feature that allows you to track the relationships between data entities, such as tables, columns, and rows, across different systems and platforms. It provides a visual representation of the data flow, enabling you to understand how data is transformed, aggregated, and used throughout the data pipeline.

Q: Why is DataHub Lineage important?

A: DataHub Lineage is essential for several reasons:

Data Quality: By understanding the data flow, you can identify potential data quality issues and take corrective actions.
Data Governance: Lineage helps you to track data ownership, access, and usage, ensuring that sensitive data is handled properly.
Compliance: Lineage is crucial for regulatory compliance, as it provides a clear audit trail of data processing and transformation.
Data Integration: Lineage enables you to integrate data from different sources, making it easier to create a unified view of the data.

Q: How do I create a lineage in DataHub?

A: To create a lineage in DataHub, you can use the following methods:

DataHub CLI: Use the datahub ingest command with the --lineage option to create a lineage from a SQL query.
DataHub Python SDK: Use the DataHubGraph class to parse SQL queries and create a lineage.
DataHub UI: Use the DataHub web interface to create a lineage from a SQL query or a data source.

Q: What are the benefits of using DataHub Lineage?

A: The benefits of using DataHub Lineage include:

Improved Data Quality: By tracking data flow, you can identify and fix data quality issues.
Enhanced Data Governance: Lineage helps you to manage data ownership, access, and usage.
Simplified Data Integration: Lineage enables you to integrate data from different sources.
Better Compliance: Lineage provides a clear audit trail of data processing and transformation.

Q: Can I use DataHub Lineage with other data platforms?

A: Yes, DataHub Lineage can be used with other data platforms, including:

Apache Spark: Use the DataHub Python SDK to create a lineage from Spark SQL queries.
Apache Hive: Use the DataHub Python SDK to create a lineage from Hive SQL queries.
Amazon Redshift: Use the DataHub Python SDK to create a lineage from Redshift SQL queries.

Q: How do I troubleshoot DataHub Lineage issues?

A: To troubleshoot DataHub Lineage issues, follow these steps:

Check the query syntax: Ensure that the SQL query is correctly formatted and follows the DataHub SQL syntax rules.
Verify the configuration files: Double-check the configuration files, including the YAML and JSON files, to ensure that they are correctly formatted and contain the necessary information.
Use the DataHub CLI or Python SDK: Use the datahub ingest command or the DataHubGraph class to create a lineage and troubleshoot issues.

Q: Can I customize the DataHub Lineage UI?

A: Yes, you can customize the DataHub Lineage UI by:

Creating custom dashboards: Use the DataHub web interface to create custom dashboards and visualize lineage data.
Defining custom metrics: Use the DataHub web interface to define custom metrics and track lineage data.
Integrating with other tools: Use the DataHub API to integrate with other tools and platforms.

Q: How do I get started with DataHub Lineage?

A: To get started with DataHub Lineage, follow these steps:

Install DataHub: Install the DataHub CLI or Python SDK on your machine.
Create a DataHub account: Create a DataHub account and set up your environment.
Run a sample query: Run a sample SQL query to create a lineage and test the DataHub Lineage feature.

By following these steps and using the DataHub Lineage feature, you can improve data quality, enhance data governance, simplify data integration, and ensure better compliance.