Back

Blockchain-based CES Solution Proposal

Published at 23rd Nov 2023

CES Backend Service using Blockchain (DLT) for Open Source ecosystem

Introduction

Recently there have been discussions about exploring the usage of Blockchain with CES. During the discussions there were some valid concerns raised. So, this proposal of solution is a step towards supporting Gaia-X and other open source communities looking for a similar solution for preparation of the standards.

 

DISCLAIMER: 

This is only a proposed solution keeping in mind the problem statement specific to Gaia-X OSS, however it can be implemented for other communities as well with the same purpose. Multiple perspectives were considered while defining this solution but it is not an approved solution from Gaia-X AISBL or the community yet. If this is approved, then we will publish another document with the updates.

 

The proposal addresses the enhancement of the credential-event-service (CES) within the Gaia-X framework. We aim to upgrade the CES backend by introducing blockchain technology to store VCs and achieve partition tolerance and high availability. This proposal outlines our strategy and tech stack for a solution to integrate blockchain along with CES backend service.

Problem Statement

The existing credential-event-service (CES) backend, which relies on Kafka, has effectively maintained data consistency and availability. However, as we advance our implementation to synchronize the Catalogue, we must consider implementing blockchain technology while addressing the challenges associated with partition tolerance and high availability.

Partition Tolerance:

To ensure that the system continues to function even in the presence of network partitions or communication failures. 

High Availability: 

To guarantee that the service remains accessible and responsive, even in the face of node failures or infrastructure issues.

Existing CES Backend

Currently we have very simple CRUD endpoints in the CES Backend. When we submit a credential event, after verification a message is produced in Kafka and also consumed in CES itself and then stored in the database. When we retrieve the credentials, they are read from the database and provided in the response.

Proposed Solution

  • For storing the credentials on an immutable ledger, we can use Hyperledger Besu for a private chain and is EVM compatible, so that we can write smart contracts for custom logic to handle the credentials.
  • The message will be consumed from Kafka by a consumer service first. This consumer service will make transactions to the blockchain for storing the credentials.
  • When reading credentials, we can use Hyperledger Besu's JSON RPC endpoints to directly read the event logs emitted by the contract from the blockchain itself.
  • Blockchain is used to ensure data integrity and transparency in CES backend service. It stores verified credentials (VCs) securely and transparently. The Express Backend API handles data input and retrieval from the Hyperledger Besu blockchain. Kafka and Zookeeper manage data streaming for reliable processing. This setup enhances security, trust, and compliance in your CES service.

 

  • Express Backend API : 

    • POST /credential-events: This endpoint will accept VCs and publish it to a Kafka topic for further processing. 
    • GET /credential-events/{id}: This endpoint will read data from the Hyperledger Besu blockchain.
    • GET/credential-events/all: This endpoint will read and return data from the blockchain.The api will return the 20 most relevant VCs or VCs per block based on your api calls.

  • Hyperledger Besu: 

    • EVM compatible private blockchain, with a minimum of 4 nodes validating transactions, using IBFT2 consensus mechanism, nodes can be scaled as per requirement.Hyperledger Besu offers a wide range of use cases from the finance, supply chain, healthcare, and blockchain industries. 
      • The Central Bank of Norway is using Besu for the CBDC sandbox & and LACChain is leveraging Besu's capabilities to foster digital innovation and financial inclusion in the Latin American and Caribbean region.
      • IBFT 2.0 (Istanbul Byzantine Fault Tolerance 2.0) is a consensus algorithm used in blockchain networks, particularly in Ethereum-based systems. It allows for dynamic validator sets, offers quicker finality, and introduces features like round-robin signing. This makes it more flexible, fault-tolerant, and economically aligned with Ethereum, suitable for various applications.

  • Consumer: 

    • Consumers will continuously listen to the Kafka topic where data is published from the POST API. Upon receiving new data, this consumer processes it, creates transactions, and submits them to the Hyperledger Besu blockchain.

  • Kafka and Zookeeper : 

    • ZooKeeper: 

      • Manages leadership election of Kafka brokers. 
      • Handles service discovery and cluster topology so brokers know the state of the cluster.
      • Tracks broker entrance, exit, and failures.
      • Determines the preferred leader for each topic/partition. 
      • Maintains a list of topics and tracks topic creation and deletion. 
      • Provides a synchronized view of the Kafka cluster. 

    • Kafka: 

      • Manages connections between producers and consumers. 
      • Handles topic logs and log partitions for data storage and retrieval. 
      • Manages consumer groups for efficient message consumption. 
      • Tracks individual consumer offsets to resume reading from the correct point.
         

CES-Blockchain-V3.png

Technology Stack

  • Kafka and Zookeeper
  • Hyperledger Besu
  • NodeJS / Express APIs
  • Consumer Service
  • Kubernetes
  • AWS

Performance and Scalability

To optimize the performance and scalability of our credential-event-service (CES) backend, we employ a strategic combination of Kafka and blockchain technology:

  1. Kafka as a Transaction Buffer: Kafka acts as a buffer between incoming real-time requests and blockchain transactions. This serves two essential purposes: it smoothens the ingestion of data into the blockchain and provides an interface that is well-suited for high-throughput real-time requests.
  2. Blockchain Considerations: Blockchain, by its nature, may not provide rapid transaction processing. Therefore, we utilize Kafka to accumulate incoming data and let our consumer service interact with the blockchain periodically. This approach minimizes the potential bottlenecks in transaction speed while allowing for efficient, sequential on-chain operations.
  3. High Availability with Hyperledger Besu: Our choice of Hyperledger Besu, deployed on a highly available Kubernetes cluster, ensures a robust and resilient blockchain environment. Leveraging cloud providers like AWS offers further redundancy and accessibility.
  4. Optimized Storage: To maintain high performance, we configure Elastic Block Store (EBS) and Elastic File System (EFS) storage with increased throughput and I/O operations. These storage systems are closely integrated with our blockchain nodes, ensuring that blockchain data is stored and accessed efficiently.
  5. Handle partition tolerance, on the blockchain side we can implement data replication and resiliency strategies such as using AWS EBS volumes or Amazon EFS for persistent data storage and using Kubernetes StatefulSets or Operators to manage stateful workloads.
  6. Load Balancing and Autoscaling: Implement load balancers and autoscaling mechanisms to dynamically adjust the consumer service's capacity in response to the number of incoming requests or messages in Kafka. This ensures that the system can handle varying workloads while maintaining responsiveness.
  7. Read Replicas for Improved Query Performance: For read-heavy workloads, we can also provision read replicas or read-only nodes within the blockchain network. These nodes are optimized for quick data retrieval and can offload read requests from the main blockchain nodes. This strategy helps maintain low latency for read operations while ensuring the integrity of write transactions in isolation.

 

By combining the strengths of Kafka and blockchain with the high availability and optimized configuration of Hyperledger Besu on AWS, the CES backend is poised to deliver exceptional performance, scalability, and resilience for handling the evolving demands of Gaia-X's data infrastructure.

 

This optimized setup guarantees a seamless experience for both real-time and batch operations, allowing for efficient data handling and retrieval in a secure and decentralized environment.

 

Proposed solutions for possible valid concerns in open source community

 

  1. Who pays for the gas in Hyperledger Besu? 
    1. In Hyperledger Besu, users have the option to perform Zero Gas Transactions, which means that transactions can be executed without requiring users to pay gas fees. This makes it convenient and cost-effective for participants using the network. 

 

  1. How can users mitigate the impact of volatile pricing in Hyperledger Besu? 
    1. With Zero Gas Transactions, users can mitigate the impact of volatile pricing since they don't have to worry about transaction fees regardless of the market conditions. 

 

  1. Which consensus mechanism? Proof of Work?
    1. Going ahead with IBFT2 consensus for now, for our use case. IBFT 2 is a consensus algorithm designed for private and permissioned blockchain networks. It ensures network security and consensus among nodes even if some nodes are faulty or malicious. In essence, IBFT 2 uses a voting mechanism where a supermajority of nodes must agree on a transaction before it is considered valid. This consensus algorithm helps maintain the integrity of the blockchain and prevents double-spending without the need for energy-intensive proof-of-work mechanisms.

 

  1. What is the advantage of using Blockchain over traditional Database?
    1. Blockchain offers a significant advantage over traditional databases by providing immutability and a more trustworthy data storage system. In a blockchain, once data is added to the ledger, it becomes nearly impossible to alter or delete, ensuring the historical integrity of the records. This immutability is achieved through cryptographic hashing ,making it highly resistant to fraud and tampering. Additionally, the transparent and distributed nature of blockchain technology increases trust among participants, as they can independently verify the data's authenticity, fostering transparency and reducing the risk of data manipulation or corruption.