The blockchain revolution has ushered in a new era of decentralization and trustless transactions, promising unparalleled security and transparency. However, as enterprises increasingly adopt blockchain technology, concerns are emerging about the immaturity of key management systems (KMS) within distributed ledger technology (DLT). This article explores the challenges enterprises face in implementing robust KMS solutions for DLT, covering best practices, pitfalls, alternative approaches, and a vision for a secure and seamless future, illustrated through insights from Hashicorp, challenges in using Hashicorp Vault, and a MakerX case study.
Best practices for key management systems
KMS are systems that are used to secure cryptographic keys and perform cryptographic secure operations such as encryption, decryption, signing, and verification. In the traditional sense, the practice of key management has been ingrained in our systems for a long time now, and its best practice principles are well understood.
The principles can be summarized as follows:
- Isolation: Keys should be isolated from the systems that use them. This is to prevent the keys from being compromised if the system is compromised.
- Access control: Strict access controls should be imperative to ensure that only authorized users can access the keys.
- Tamper resistance: The KMS should be resistant to tampering. This is to ensure that even in the event of system tampering, the cryptographic keys remain uncompromised.
- Auditing: The KMS should be auditable. This is to ensure comprehensive oversight of key usage, confirm that the keys are being used correctly, and make it possible to forensically determine how they were used when investigating potential breaches.
- Interoperable integration APIs: The KMS should adopt standard and agreed protocols for integration to facilitate easy, consistent, and secure integration. This also helps with compliance and security regulatory requirements.
- Hardware security: Solutions should involve hardware security modules (HSM) because keys that see memory run-time space are prone to being compromised, whereas hardware-based solutions are more secure.
The state of KMS in Web3
An absence of seamlessness and security
In contrast to the conventional disconnection of traditional KMS from their usage context, the current user experience in Web3 falls short of seamless and secure transaction signing. In everyday applications like sending a WhatsApp message, users benefit from encryption schemes and digital signatures seamlessly integrated behind the scenes. Web3, unfortunately, has struggled to deliver a similarly intuitive and transparent experience.
Moreover, the current paradigm for end-user-facing Web3 wallets necessitates that users see and back up the “recovery phrase” (essentially their private key) to keep themselves resilient to disaster. The fact that users can export their private key coupled with the fact that the private key is rarely kept in a secure (ideally) hardware enclave means that the current experience fails the principles of isolation, tamper resistance, and hardware security.
A “copy-paste” hell
In the Web3 industry, there's a heavy "copy-paste" approach to techniques and standards. Additionally, in the early days of Bitcoin, cryptography choices like secp256k1 necessitated doing a lot of key management in a customized, in-memory manner. At that time, traditional devices, hardware, and KMS lacked support for the chosen elliptic curve.
This has given rise to a proliferation of custom key management solutions and the emergence of Web3 hardware vendors that have appeared without adopting the traditional principles of key management and hardware-based security.
Lack of open hardware wallets
The immaturity of the Web3 industry becomes apparent when businesses and wallets lack a common standard for integrating with hardware security devices in an interoperable way that helps them meet the compliance and security of standards like ISO, FIPS, and PCI.
Currently, each vendor operates with its proprietary software development kits (SDKs), APIs, and operating system-specific libraries. This fragmentation poses a substantial burden in terms of cost and time for businesses and developers, particularly when aiming to develop vendor-agnostic solutions.
Traditionally, hardware manufacturers (e.g., HSM, security key, mobile phone) adhere to open standards such as PKCS#11, KMIP, etc. This adherence ensures that devices supporting these standards can be seamlessly integrated using the same APIs and SDKs. Embracing a standards-based approach to blockchain is essential for the industry's maturation and technological advancement. The enterprise adoption of blockchain serves as a natural driver for the establishment of such standardized practices.
No KMS isolation
A prevalent security concern within Web3 wallets lies in the violation of the fundamental KMS security principle of isolation. In many instances, wallets operate without adequately isolating key material, with in-memory wallets storing private key information and performing cryptographic operations within the same run-time space as the application/wallet. There's no logical separation of memory, processes, and permissions between components that handle keys, which poses a great security risk. A more robust approach necessitates the implementation of a logical separation to enhance the overall security posture of Web3 wallets.
Thus, the current landscape doesn’t meet the desired user experience or essential security posture. Something has to change to meet the needs of enterprises.
A proposal: How enterprise blockchain KMS should look
An alternative approach rooted in KMS principles
To effectively assist serious businesses and enterprises in adopting DLTs, such as Algorand, it is crucial to implement a key management approach that aligns with the following principles:
- Multi-tenancy: Unlike most Web3 SDKs and wallets tailored for single users, an enterprise-grade solution must accommodate authentication for multiple users and even unattended system services that might perform automated cryptographic operations.
- Isolation: Designing the system with a clear separation of private key material from running applications is essential to significantly reduce the risk of compromise. This principle is often broken in Web3 due to the direct use of SDKs like Web3.js, ethersjs, algosdk, etc. within the same runtime space as the application.
- Resilience: The key management system should be reliable to enterprise standards, maintaining high uptime. Additionally, robust mechanisms for disaster recovery (DR), including the recovery of key materials, must be in place to prevent data loss.
- Security: Implementing standard enterprise security practices, such as defense in depth, least privilege, access control, encryption in transit and at rest, auditing, etc., is imperative to fortify the overall security posture of the KMS.
- Compliance: The KMS should adhere to any required compliance standards, including but not limited to ISO, FIPS, etc., ensuring that it meets industry-recognized benchmarks.
- Open: Choosing an implementation based on open standards streamlines integration processes, making it more accessible and adaptable for various applications and environments.
HashiCorp Vault for enterprise blockchain scenarios
In order to follow these principles, we recommend that enterprise blockchain scenarios make use of proven, off-the-shelf KMS rather than opting for bespoke approaches.
Namely, this includes solutions like hardware security modules (HSMs), Fireblocks, CloudHSMs, Azure Key Vault, or proven software solutions like HashiCorp Vault. While using hardware enclaves is ideal, considerations such as cost, resiliency, and a lack of support for a broad signing algorithm often rule out HSMs for transaction signing. It's crucial to note that if an HSM doesn't support the signing algorithm natively, it defeats the purpose of using the HSM, as the secret key material would have to leave the enclave.
Given these considerations, we recommend HashiCorp Vault as an ideal solution for enterprise blockchain signing scenarios when configured and architected suitably. Here’s why:
- Proven track record: HashiCorp Vault has demonstrated its effectiveness over many years of use in traditional cryptography scenarios, establishing itself as an industry-leading solution with documented case studies.
- Open source and audited: HashiCorp Vault is open source, heavily audited, compliant with many security standards, and has built-in audit devices and telemetry.
- Industry cryptography standards: While proprietary, the HashiCorp Vault heavily relies on industry cryptography standards and provides a consistent interface atop these standards.
- Tamper-proof security model: HashiCorp Vault offers a tamper-proof security model, enhancing its overall reliability and security.
- Wide cryptographic algorithm support: The system natively supports a broad spectrum of cryptographic algorithms and engines, so the key material doesn’t need to leave the Vault enclave.
- Resilient deployments: HashiCorp Vault facilitates resilient deployments through mechanisms for high availability and encrypted backup/restore processes.
- Isolation of key material: Key material is isolated within the vault, allowing for key creation within the vault itself. With correct configuration, administrators do not have access to secret material, and stringent access policies restrict the use/export of that material.
- Advanced functionality: HashiCorp Vault offers advanced functionality that can be used to enable secure individual user access to keys through identity-based access management. Additionally, it supports wrapping all secret material with a FIPS-compliant HSM.
How Hashicorp Vault works
Hashicorp Vault is a popular open-source KMS that is used by many enterprises. It supports a wide range of cryptographic algorithms and is compliant with many security standards.
The list of features that are available out of the box is indeed impressive and includes secret management and rotation, encryption/decryption and associated key management, signing capabilities, identity-based access management, comprehensive audit logging, integration with cloud services and HSMs, multi-tenancy, high-availability and advanced recovery mechanisms.
Shamir’s secret sharing
Shamir’s secret sharing, a cryptographic algorithm, facilitates the division of a secret into shares. These shares can be distributed among different individuals, and by meeting a specified threshold (e.g., 2/3 present, 3/5 present, etc.), the original secret can be reconstructed.
Vault initialization
Vault initialization is a process where the root key is generated and split into shares based on a defined threshold using Shamir’s secret sharing. These shares are then distributed to the unsealers, which should be distinct individuals or trusted organizations. This is a key mechanism that prevents a single admin user from being able to access secret key material.
Vault sealing and unsealing
At rest, the Vault is sealed (encrypted) and requires unsealing for use. This is accomplished by providing the threshold of Shamir’s secret shares that were generated during initialization. This process can be executed manually by multiple people or automatically through multiple API calls.
Access policies and roles
The Vault has a very granular access control system that enables the definition of policies and roles. This can be used to restrict access to secrets and operations and to disallow the unauthorized export of secret material from the Vault.
Transit Engine
The Transit Engine, a Vault plugin, must be enabled to execute cryptographic operations such as encryption, decryption, signing, and verification on data in transit (not stored, e.g., an unsigned blockchain transaction). It supports a wide range of algorithms, including most of the key blockchain signing algorithms, and is compliant with many security standards. Out-of-the-box supported algorithms include AES-GCM, ECDSA, ED25519, RSA, and SHA2, with a plugin ecosystem for additional options.
Challenges and considerations of using Hashicorp Vault
While Hashicorp Vault has many commendable features, it’s not a solution to deploy lightly. It’s a highly configurable and flexible solution that requires a fair amount of domain knowledge to set up and secure correctly.
This can be partially offset by paying for either the HCP Vault (hosted) or Enterprise tiers. The HCP Vault tier ensures reliable hosting of the core Vault, while the Enterprise tier provides access to convenient plugins that make it easier to set up an enterprise-grade cluster, in addition to hosting.
While this helps you get a reliable Vault cluster, additional work is needed to ensure it’s configured correctly. Furthermore, the pricing structures of the hosted or enterprise tiers might pose cost challenges for smaller enterprises or prototype use cases. Given that HashiCorp Vault is open source, there's an option to build your own cluster without incurring licensing fees. However, it's crucial to note that this entails substantial work in creating an automated, highly available cluster. Proficiency in configuring and operating HashiCorp Vault clusters is strongly recommended for this approach.
To take the base Vault cluster and build a reliable enterprise-grade solution for signing blockchain transactions, the following additional steps must be taken:
- Consideration of how Shamir’s secrets are handled to avoid a single administrator being able to gain access to the private keys as the secret shares must be assembled on a single node for private key reconstruction before signing
- Automation to ensure high reliability with resilience to data center outages, including auto-unseal (without compromising the first point), resilient cluster leader handling, and effective availability zone handling.
- Configuration of administration and configuration accounts, as well as end-user identity providers (ideally automated).
- Creation and configuration of transit keys for blockchain transaction signing, and then the development of (ideally automated) access policies on these keys to prevent unauthorized use or export.
- Implementation of operational processes and integrations for telemetry, auditing, and disaster recovery (backup/restore) to ensure a comprehensive approach to system monitoring and data integrity.
- Use of APIs to properly propagate the end-user identity and correctly pass the encoded transaction for signing, using the resulting signature before passing the signed transaction to the blockchain.
Algorand Transaction Building
The integration of HashiCorp as a KMS for Algorand will not use the traditional SDKs and therefore we will need to understand how to craft the correct models, perform encoding and select the appropriate signature scheme.
Model
The first step in building an Algorand transaction is to craft the model that will eventually be encoded and signed. When looking for the structure in which these should be built, the following format is available in most documentation:
However, this model is very specific to algosdk and goal, not to the protocol. It turns out that directly encoding the unsigned version of this transaction and attempting to sign it with a KMS wouldn't be validated by a node. This arises due to the following key differences between the expected model and the one that is actually signed:
- Only the contents of txn need to be encoded.
- Properties need to be sorted alphabetically.
- For the JS Algorand SDK, the encoding is done through msgpack, which is a fork of the standard msgpack. Find it here: https://www.npmjs.com/package/algorand-msgpack.
- “snd” and “rcv” are represented as encoded addresses, but they should be signed as the corresponding binary public keys.
- “gh” is the genesis hash, but it needs to be decoded from base64 and then encoded as an array of bytes.
Encoding
Before encoding the contents of txn, the fields: amt, fee, fv, gen, gh. lv, note, rcv, snd, type need to be alphabetically ordered.
Algorand cryptography
In Algorand, the standard Ed25519 signature scheme is used to sign and verify transactions. The picture below shows the regular Ed25519 signature scheme and the steps required to generate and sign.
If you try to integrate a regular KMS or HSM to perform signing instead of one of the SDKs, some might ask you to specify which hashing function you want to use as part of the signing process, such as Hashicorp Vault. You will need to specify SHA512 as the hashing function of choice.
The algosdk does have a function, bytesToSign, to output the raw bytes that need to be signed. However, this method requires constructing the transaction as outlined earlier and importing all necessary dependencies. This method lacks a mechanism for validating the correctness of the generated bytes or offering guidance on their decoding.
These types of SDK hoops aren’t unique to Algorand ecosystem tools. There is a certain level of immaturity in the Web3 industry when interfacing with more established KMS, where the custom APIs and protocols become a friction point.
Furthermore, importing large SDKs to only use a small portion of them creates problems for enterprise-level applications, such as application package bloat, unwanted third-level dependencies, unaudited packages, and a lack of fine-grained control over operations and integrations.
To address these challenges, dedicated packages, and specialized tools have become increasingly appealing for such integrations. An example is algo-models, a lightweight package dedicated to performing correct modeling, schema validation, and encoding in a format that can be consumed by any KMS. This approach holds promise, and we would love to see it propagate more broadly in the industry.
A case study: MakerX enterprise signing platform
Algorand Foundation partner MakerX, the implementation partner behind AlgoKit, has recently built a fully automated, enterprise-grade key management platform based on Hashicorp Vault and algo-models to craft and encode transactions. This platform aims to facilitate secure Algorand transaction signing.
The impetus behind designing the system was driven by the specific needs of one of MakerX’s clients. The client sought regulatory-compliant custodial accounts for an application in the financial services space. In this case, the application would be governed by a custodial license granted by the Australian Securities and Investments Commission (ASIC). The private keys would potentially secure millions of dollars worth of assets on behalf of end users.
The design of the platform was guided by three key architectural principles, ensuring it met the stringent requirements for a secure custodial solution:
- (Semi-)Decentralized key access: No single person or administrator would get access to the private keys. Multiple parties from different (semi-trusted) organizations would have to come together to gain access.
- Resilient recovery: Given the keys would be responsible for assets worth a significant amount, the solution would need to be resilient to failure, and in a disaster recovery scenario, the keys would need to be recoverable while still conforming to the first principle.
- User-present key use: A key would only be used when an end user was provably present. A short-lived access token, sourced from an identity provider and accessible only to a specific user, would serve as the sole point of access to a key. The key material itself would be restricted from leaving the enclave, allowing for the audibility of key usage.
The high-level architecture of MakerX’s platform looks something like this:
In the end, MakerX built an impressive production-grade platform that is entirely automated while conforming to these principles. This is a great example of enterprise key management for blockchain systems done right. If you're considering deploying a similar platform for your specific use case, it may be worth contacting MakerX for further information or discussion.
Concluding thoughts
There are successful KMS integrations available that adhere to traditional KMS principles in a Web3 setting, as demonstrated by MakerX and its use of HashiCorp Vault to implement regulatory-compliant custodial accounts. This marks the first successful integration of Algorand's technology stack by an enterprise-grade application with strict compliance needs. The integration demonstrably helps the business meet those regulatory requirements. In our opinion, this signals a new level of maturity that Web3 has been lacking for years, where the norm has been to ignore previous patterns, standards and techniques when it comes to integration, key management and security.