As Ibrar Ahmed noted in his blog post on Transparent Database Encryption (TDE). PostgreSQL is a surprising outlier when it comes to offering Transparent Database Encryption. Instead, it seems PostgreSQL Developers are of the opinion that encryption is a storage-level problem and is better solved on the filesystem or block device level. I do not share that opinion, and according to a poll I did late last year, many PostgreSQL community members don’t either.
Does PostgreSQL Need TDE (Transparent Data Encryption)? #postgres #security
— Peter Zaitsev (@PeterZaitsev) October 24, 2022
There are two reasons TDE implemented inside the database is essential—technical reasons and non-technical reasons. In my opinion, engineers tend to focus on the technical side and stop on “technical solutions exist”, while in practice, what really matters, is if an organization is able to implement the solutions offered. This is where organizational issues (especially politics) play a big role, especially in large organizations.
For example, only a fraction of Oracle (and other legacy databases) workloads that could be migrated to PostgreSQL are actually migrated, even if it offers substantial cost savings and additional benefits, all because of organizational resistance.
But first, where does the need for encryption really come from? On a high level, some of it comes from Technologists who understand unencrypted sensitive data is not a good idea. Increasingly, though, the industry is not relying on engineers having common sense but rather on defined standards that companies need to comply with (HIPAA, PCI DSS, SOC 2, ISO 27001), etc.
Compliance requirements are typically written in rather general terms, and it is up to the compliance team (and compliance audit team) to translate these to technical requirements, many of which would prefer (or even require) TDE.
Technical reasons for Transparent Database Encryption
Compared to storage-level encryption, encryption on the database side offers multiple benefits, including:
- It offers more flexibility and granularity in the encryption of database objects (what is encrypted, what is not, and with what key).
- If database files are copied or otherwise exposed in their raw form, exposure does not happen. For example, physical backups stored in open S3 buckets are relatively common exposure vectors.
- It does not require encryption capabilities on the storage level to provide data security.
- It adds defense in depth. Even if the storage level encryption exists, having TDE plus storage level encryption gives an added layer of security.
Organizational (non-technical) reasons for TDE
Even if you found a way to deal with all the technical reasons you prefer encryption inside your database engine (TDE), you may be facing additional friction in your organization choosing this path. This might be a challenge with the security/compliance team (as I already mentioned above) or require additional coordination with the team responsible for storage to ensure proper encryption of original data and all copies (including backups).
While these will look trivial for startups and small companies where all of those are single-person or well-connected teams, you may be surprised by how expensive those things are in some large organizations!
Encryption in the cloud Database as a Service (DBaaS) space
You may argue that storage-level encryption seems to be good enough for cloud vendors. AWS, for example, offers encryption in RDS, which appears to be good enough for most of its customers! However, things are a lot different in RDS environments; it is a rather restricted environment where many technical and organizational aspects are taken care of. For example, if you have created an encrypted instance, backups will be automatically encrypted, too, eliminating the possibility of a “mistake.”
State of TDE in PostgreSQL
Interestingly, the challenges of implementing Transparent Database Encryption in PostgreSQL are not just technical but organizational too. In my memory, there have been several implementations offered for inclusion in PostgreSQL, but none made it through. The Cybertec encryption patch existed for many years, and there was also work by NTT submitted a few years ago. Highgo seems to have worked on TDE, too, and there were talks at PgCon in other years. What does not happen, though, is a feature merge into PostgreSQL Core.
You may argue the PostgreSQL Core is conservative by design, and this is what made PostgreSQL such a stable product. I think, though, one needs to maintain the balance between innovation and stability; shifting too much to one side risks the long-term project future.
Note I do not argue that including full TDE support in PostgreSQL Core is the right idea; it might be at this stage, we need innovation to happen through competition of several ideas and several implementations. What the PostgreSQL Core might highly benefit from is a pluggable storage interface, as this may not only implement encryption but also store data somewhere else than a locally mounted POSIX file system, such as what Neon is doing.
Beyond Transparent Database Encryption in PostgreSQL
While I believe Transparent Database Encryption in PostgreSQL is important, I think it is just an illustration of a bigger question. Is technical governance in PostgreSQL designed to maximize its success in the future, or is it more about sticking to the approaches that helped PostgreSQL reach current success levels? For a project of such scale and influence, there seems to be surprisingly little user impact on PostgreSQL Governance. The PostgreSQL Core Team consists of “seven long-time community members with various specializations” rather than having clear electable positions, as many other open source organizations do. The development process in PostgreSQL is based around a mailing list rather than more modern and organized issue tracking and pull-request-based development workflows. Interested in PostgreSQL Bugs? There is no bugs database that allows you to easily see which bug is confirmed and what version it was fixed in a user-friendly way. Instead, you need to dig through the bugs mailing list.
Do not get me wrong, PostgreSQL is a fantastic database, and it rightfully was Database of the Year more than any other database. Yet, I believe there are some opportunities to make PostgreSQL even more “future-proof” and be even more successful in the future.
Percona Distribution for PostgreSQL provides the best and most critical enterprise components from the open-source community, in a single distribution, designed and tested to work together.
I couldn’t agree more with you.postgresql still has very difficult problems until now，eg xid32 ，No connection pool