Updates on HEIR, the homomorphic encryption compiler project

On 2026-08-14 I published an article on the Google Security blog with an update on HEIR, our homomorphic encryption (HE) compiler. This is a companion article, in which I have no limits on word count or jargon, and I can feel free to be honest. So strap in.

Assuming you won’t read the linked corporate blog post, HEIR is a compiler that converts an input program to a program that operates directly on encrypted data. The guarantee of homomorphic encryption is that, assuming you haven’t cracked the cryptography, at no point does the computer running the program get even a single bit of information about the cleartext data used to generate the encrypted inputs. No information about the inputs, outputs, or any intermediate values.¹

The blog post focuses on HEIR’s ability to compile pre-trained ML models, and gives four examples of small, but nontrivial models that it can compile. Hence, homomorphic encryption can enable services to provide perfectly private inference. I’ll try to say more about when and where this is useful later in this article. First I wanted to give a more concrete sense for how HEIR works in the context of these examples, and outline (my view on) the project’s roadmap for the future. I won’t do a deep dive on HEIR’s internals by any means, since that would make the article too long. Give me a shout if you want that, but there are plenty of docs to read through at heir.dev and you can see a recent (fast-paced) talk I gave at ASPLOS this year.

Table of Contents:

The repo behind the blog post

The blog post ends with a list of examples compiled with HEIR. Those examples point to a GitHub repository² that you can clone and run yourself. The biggest hurdle is installing bazel, and then bazel hermetically manages everything else.³

Some simple runtime comparisons

The simplest and fastest example to try is the credit card fraud detector. This is a simple three-layer feed-forward network with sigmoid activations, trained on a Kaggle dataset. The linear layers have dimensions 128, 64, and 2 (the last being the logits for the two classes, fraud and not-fraud).

You can run the basic example in one line:

bazel run -c opt //demos/cc_fraud/lattigo:evaluate_fhe

This command will compile the (pre-trained, checked-in) cc_fraud model to the Lattigo backend, and then run it on a sample input. The command above outputs:

Loading test row 0 from /home/jeremy/fully-homomorphic-encryption/demos/cc_fraud/data/test_rows.csv...
  Took 83.226µs
  Expected label (is_fraud): 0
  Feature vector size: 82
  First 5 features: [-0.31676582 0.85089076 -0.40874073 -0.1833772 -1.7155787]
Configuring Lattigo context...
  Took 2.164051998s
Encrypting input features...
  Took 19.430069ms
Running preprocessing...
  Took 573.537941ms
Running FHE evaluation (preprocessed)...
  Took 2.020821739s
Decrypting output...
  Took 488.237µs
Decrypted logits: [16.464235 -16.781752]
Predicted class: 0
SUCCESS: Predicted class matches expected label!

The central point here is that the evaluation of the model on encrypted inputs took about 2 seconds on a single-threaded CPU.

Compare this to the same execution on cleartext inputs, noting that this is the latency of a single inference, so it doesn’t benefit from amortization. (This requires fetching and encoding the original dataset, which is explained in the README; I’ll skip that part here).

$ bazel run -c opt //demos/cc_fraud/cleartext:evaluate_cleartext
Loading model from: demos/cc_fraud/data/mlp_fraud_model_sigmoid.pt

Evaluating Credit Card Fraud Sample Index: 0
True Label:      0 (LEGITIMATE)
Predicted Label: 0 (LEGITIMATE)
Fraud Probability: 0.000000
Result:          CORRECT
Latency:         0.5233 ms

Anyone who has heard of HE may have heard that it is slow, but I want to pause here to compare this (single-threaded CPU, non-amortized!) runtime: 2 seconds for HE inference vs 0.5 ms for cleartext. This is a 4,000x slowdown, and the computation involves two matrix-vector products (where the matrix is not private), with two evaluations of a sigmoid function.

There are many caveats to this demo worth briefly noting:

• This model is small enough that the private information fits in a single (CKKS) ciphertext. Inputs tensors with more than, say, 32k elements will require multiple ciphertexts and naturally more overhead.
• This model is small enough that bootstrapping (the slowest part of HE) is not required.
• I did not include a simulated network overhead, nor did I include the per-user one-time setup required to generate and upload the relevant key material that makes HE work.
• The server is doing a decent amount of (one-time, model-specific) pre-computation in the “Running preprocessing” step.

The other examples in the repo are more complex, and hence have longer latencies and worse overhead vs cleartext (and a memory requirement of 60-90 GiB). In particular:

• network_anomaly, an ensemble of auto-encoders: 30 seconds for inference.
• criteo, a recommender model adapted specifically for HE: 5 minutes for inference.
• hotword, a 10-layer convolutional network: 20 minutes for inference.⁴

This sounds bad, but remember it’s single-threaded CPU execution. Our colleagues working to integrate HEIR with GPUs have reported that the criteo workload runs in ~500ms on a single GPU (similar to an H100). Compare that to the criteo/cleartext:evaluate_cleartext demo which runs in 10ms, and you’re down to a 50x slowdown (again, the baseline is non-amortized CPU execution). That work wasn’t able to make it into the Google blog post, but they are compiled by HEIR with some pending upstream PRs. My point is that the execution times are continuing to improve, and for some small problems they could be called reasonable if you squint. And this does not even breach the topic of HE accelerated by FPGAs and ASICs, which are even more promising performance-wise.

So instead of the Google corporate blog post showcasing raw performance, it was meant to showcase the expressiveness of HEIR: it can compile a lot of models, and the performance on some of them is decent.

As far as showcasing features, the repo also shows how one might use HEIR to:

• Target and compare multiple HE backends (the repo has Lattigo and OpenFHE as examples, try bazel run -c opt //demos/cc_fraud/openfhe:evaluate_fhe.
• Explore per-layer timing: HEIR has the ability to insert debugging callbacks into the compiled program, try bazel run -c opt //demos/cc_fraud/lattigo:evaluate_fhe_timing. In that command you’ll see the first linear layer takes the majority of the runtime, ~1.2 seconds out of 2 seconds total.
• Explore an incorrect inference: Similar to timing, the debug callback can decrypt ciphertexts and inspect them for correctness or precision loss. Try bazel run -c opt //demos/cc_fraud/lattigo:evaluate_fhe_debug and you’ll see output that shows that by the end of the inference, the use of HE has caused about 2 bits of precision loss compared to the cleartext model. This precision loss can be tuned with compiler flags and trades against performance. Unfortunately, navigating that trade-off requires HE expertise.

Getting a model into HEIR

The process of getting a pre-compiled model to be something that HEIR can process is not yet automated. The main two constraints are:

1. You need to be able to convert the pre-compiled model to MLIR, which is the intermediate representation that HEIR uses to represent programs. Many ML frameworks such as PyTorch and JAX have tools to convert to MLIR.
2. You need to manually annotate your program with HEIR-specific annotations that say (a) what inputs to the inference are secret and (b) what are bounds on the input ranges to each activation function.

The demos in the repository show how to do this for PyTorch, and I’m working with the maintainers of torch-mlir to add a feature that will enable me to automate this (given a validation set to use to estimate ranges).

That said, much of the early stages of the compiler pipeline (recognizing activations, fusing linear layers, etc.) is based on how torch-mlir exports models to MLIR, so if we want to add ONNX or JAX support (both have great MLIR exporters), it will likely not work out of the box just yet. Moreover, we don’t even have complete coverage of torch operators yet. A lot of the details of how to support linear algebraic operators in HE are both tricky and active research topics (for MLIR enthusiasts, we don’t support linalg.generic in full generality).

Invoking the compiler

This is a bit of a tangent, but if you look at the build rules for the examples in the repo, you’ll see some slightly messy calls to a macro that invokes the compiler.

load("@rules_heir//heir:lattigo.bzl", "heir_lattigo_lib")

HEIR_OPT_FLAGS = [
    "--annotate-module=backend=lattigo scheme=ckks",
    "--torch-linalg-to-ckks=min-slot-count=8192 greedy-level-budget=15 greedy-modulus-switch-after-mul=true experimental-disable-loop-unroll=true first-mod-bits=30 scaling-mod-bits=24",
    "--scheme-to-lattigo",
]

heir_lattigo_lib(
    name = "fraud_model_lattigo",
    go_library_name = "fraud_model_lattigo",
    heir_opt_flags = HEIR_OPT_FLAGS,
    importpath = "fully_homomorphic_encryption/demos/cc_fraud/lattigo/fraud_model_lattigo",
    mlir_src = "//demos/cc_fraud/data:model_annotated.mlir",
    split_preprocessing = True,
)

go_binary(
    name = "evaluate_fhe",
    srcs = [
        "evaluate_fhe.go",
        "utils.go",
    ],
    data = [
        "//demos/cc_fraud/data:test_rows.csv",
    ],
    pure = "on",
    deps = [
        ":fraud_model_lattigo",
        ":fraud_model_lattigo_utils",
        "//demos/common/go/pathutils",
    ],
)

What’s going on here is that HEIR’s interface is a lot more like LLVM than clang. There are two binaries, heir-opt and heir-translate, which handle running compiler passes and codegen, respectively, matching LLVM’s opt and translate. Those two binaries are wrapped into bazel rules, which I published as rules_heir, and then further wrapped the rules in macros that correctly stitch together the optimizer and codegen binaries, and wrap the results into a cc_library, go_library, or rust_library, as appropriate to the chosen backend.

One nice aspect of this is that rules_heir uses pinned binaries, and can be pointed to a nightly or custom HEIR release. Still, this is less of a clear user story than something like clang, which has a consistent interface and limited exposure of the underlying LLVM kitchen sink.

Our goal is to eventually make one (or multiple) clang-style products that have a narrower scope (e.g., only supporting one frontend) and conceptually simpler flags (like -O2). For frontends like JAX/PyTorch, this will probably be wrapped in a Python library.

Speaking of Python libraries! We do have a heir_py frontend library. It’s completely unrelated to ML at this point. It actually compiles a (very limited) subset of Python bytecode (using numba’s frontend) to homomorphic encryption, and then runs HEIR, generates OpenFHE C++ code, compiles and links that with clang (the slowest part!), and then loads the machine code back into a Python module which the user can call like a function.

To be clear, we haven’t used the Python frontend for anything particularly useful. It still needs a lot of features to start being useful, such as preserving numpy ops as MLIR linalg ops that HEIR understands.

Who wants HE?

Now I’d like to take a step back and pontificate a bit about who wants HE. There has been a lot of discussion in the HE community about this, and in particular the question has been raised: what is HE’s “killer app”?

Most academics will tell you the answer is “private LLMs.” I will reserve my opinion on that topic for dinner parties, but suffice it to say that two things are currently true: First, most if not all applied HE work in academia is currently focused on making encrypted LLM inference faster. Second, the SOTA is still very far away from being practical. In particular, latencies are measured in the units of seconds per token, and all solutions I’m aware of require a round-trip to the client to decrypt the output from one inference and prepare it for the next inference step. And this is including hardware acceleration (e.g., 8 GPUs). I’m also not an expert on LLM architectures, so I don’t have a strong sense for the current barriers and how likely they are to succumb to additional research efforts.

My focus at Google has instead been on a lower-hanging fruit: what applications can take advantage of the HE capabilities we have today? In that discussion, I have landed on the following soft heuristics, which shed some light on the remaining challenges to make HE truly practical. I will again restrict myself to private inference here.

Models from 2020

While the latency of HE has improved significantly, models with billions of parameters are still out of reach for practical applications. And in particular, this rules out any transformers that are big enough to be useful.

That said, I have spoken with engineers and scientists at Google who have informed me that, despite their efforts, their domain has NOT yet seen any benefit from applying transformers. As such, the core complexity of their model architectures has not changed significantly since roughly 2020.

Convolutions are still king in many domains. Or in other words, there are still useful tasks that don’t need huge models. And so even if a large model could be used, if privacy is important enough, a smaller one might be acceptable to get fast HE.

Critical privacy for both parties

While I aspire to a world in which HE can be applied without a critical need for privacy, we’re not there yet. So for HE to be useful, the privacy of both parties must be critical.

I say “both parties” for the following reason. The standard privacy protection one expects when discussing private inference is sensitive user data. If my email service can’t see the text of my emails, I am spared from the risk of data leaks, insider threats, exposure to warranted searches, etc. The standard way to protect user privacy in these situations is to do all the work locally on the user’s device. In late 2024, Google Maps changed its location history feature to be fully on-device, at least partly in response to US police geofence warrants.

But for private inference in particular, the trained model is also sensitive data of the service provider. Shipping a model to a device, even one with a secure enclave, risks exposure of the model weights to competitors. For use cases like biometric authentication, leaking the model also adds a security risk; attackers can engineer attacks against the production model in an unrestricted environment. I’m not an expert on hardware-based security, but some of my colleagues make a living breaking them, and the way they talk about the topic doesn’t inspire confidence in secure enclaves.

I have also been led to believe that there are use cases where keeping a proprietary model secret exceeds the importance of user privacy, because the user data in those situations is not considered sensitive. In these situations, “user privacy” is a bonus on top of protecting IP. That’s not as romantic as user privacy, but hey, if it spurs further investment in HE, I’ll put up with it to get closer to the ideal.

Not too many simultaneous users

As latency becomes less of a bottleneck for HE, other bottlenecks naturally emerge. The biggest one I see is key management.

The mathematics that makes HE work requires special “evaluation keys.” These keys are, quite literally, encryptions of the user’s secret key (or something derived from the user’s secret key). Ignoring for a moment the added assumption of circular security, what matters for practical HE systems is:

1. Evaluation keys are unique to each user of the service.
2. They scale with the program size in some sense (e.g., doing larger matmuls in HE requires more evaluation keys than a smaller matmuls).
3. The biggest examples from the HEIR demo repository need ~40 GiB of evaluation keys. SOTA LLMs in HE need hundreds of GiBs of key material.

So if you had a million-user service with the 40-GiB-per-user use case, you’d need 512x20T hard drives just to store the key material. At ~500 USD per drive (just checked Amazon, so that’s a consumer price), you’re asking for 250k USD extra storage costs. I’m sure the price of RAM needed to serve a decent QPS is going to be worse.

And more, if you are using GPUs to accelerate the HE, you have the added problem of getting this 40 GiB of key material from disk to the GPU when a user makes a request. My napkin math says this should take roughly hundreds of milliseconds–the same order of magnitude as the entire HE computation. And during the time that you’re filling up the GPU’s memory with keys for one user, you can’t serve other user requests.

To be fair, a decent chunk of the RAM requirement consists of preprocessed model weights, which are model-dependent and not user-specific. And moreover, most of the key material required is to support bootstrapping, so there may be good HE applications (like Apple’s) that can use tricks to avoid needing to bootstrap. Maybe in the best case we’d only need 1 GiB of key material. So maybe you could fit enough keys for 50 users on a beefy datacenter GPU. Still, you can see the systems bottlenecks here.

And I don’t think this is getting much better. Newer HE schemes that promise better latency tend to require larger key material, not smaller. As such, until we (the HE community) finds a way to reduce the memory requirements of HE by an order of magnitude, I am only going to be looking for applications where there aren’t too many simultaneous users.

The server shouldn’t need to see the output

A lot of people I’ve talked to who want to use homomorphic encryption eventually realize that, actually, the server wants to learn one bit of information about the user’s underlying cleartexts. For example, say we want to authenticate a fingerprint without having to store the user’s biometrics. HE would prevent the server from seeing the authentication result, and so it can’t act on that.

One basic idea for dealing with this is to send the encrypted authentication result back to the user, have the user decrypt it, and send the result back to the server. The obvious downside to this is that you can’t trust the client to honestly report the authentication result!

To make this work, you would need the client to generate a zero-knowledge proof that the value they decrypted was actually the result of running the decryption routine with the provided ciphertext and the user’s secret key. It has to be zero-knowledge because the server cannot learn anything about the user’s secret key. While I haven’t done this myself, I have heard it is both possible and there are still some challenges in scaling it to HE schemes like CKKS. And of course, it further eats away at a latency budget.

As such, near-term applications of HE are probably going to have an easier time making it to production if the application does not require the server to see the computation result.

So what applications meet these constraints?

Some ideas that seem to hold water:

Remote diagnostics. Say you run a factory and you have some expensive machine you bought from a manufacturer. You want to let the manufacturer diagnose the machine to recommend maintenance and identify problems, but the details of what you’re using the machine to make (say, manufacturing volume) or the settings of the machine itself would be somewhat sensitive financial information if it were leaked to a competitor. However, diagnostic reports are batch jobs that the server need not see the output of, there aren’t millions of factories, diagnostics are a premium service the manufacturer would want to charge for, and they are probably not that complicated of models.

I have heard rumors that someone is actually using HE for this, but I have no publicly verifiable proof. Our network intrusion detection demo is similar in nature, where the contents of the packets are the sensitive data.⁵

B2B analysis: Similar to existing applications of private set intersection at Google, business-to-business applications check a lot of boxes. Both businesses are privacy needs, but there are only two users, so key material is not an obstacle. But there is enough trust for the two parties to share the output, provided the computation is producing some set of business metrics relevant to both parties. And business analytics and metrics reporting are usually quite latency tolerant. I recall talking to one HE practitioner who said, “if the client only needs a report generated once a month, then who cares if the HE computation takes a week to run?”

Specialized biometrics situations: I think biometric applications like face and fingerprint recognition are within the capabilities of current HE techniques, with the exception of the key material problem. But there are specialized situations in which extra latency should be tolerated, such as in-person account creation or account recovery procedures (think of a new employee on-boarding day, or losing your security badge). These events are infrequent enough that the system wouldn’t be encumbered by high QPS needs. And I think it is generally a good idea that employers do not have databases of employee biometrics.

Some notes on the HEIR roadmap

I’ll close with a few notes on where HEIR is going next.

Crypto in the compiler

You may have noticed all the HEIR backends are currently external library APIs in a high level language. This is mainly because, when we started HEIR, hardware acceleration work was still in its infancy. Moreover, people building accelerators (and GPU libraries) needed an easy way to write larger applications that use their libraries for benchmarking purposes. These often mirrored some existing library API (like OpenFHE), and then injected their own code at some level, such as scheduling a key switching op on a GPU.

So we decided HEIR should meet them in the middle. Start by targeting these popular library APIs, and get the frontend of the compiler pipeline working well. At the same time, we have been slowly building out a true “backend” of the compiler, which involves implementing the HE cryptography directly as compiler passes. With that in place, and with our multi-layer design enabled by MLIR, we can compile a program down to a library API (high-level HE scheme operations), or to a lower-level modular polynomial arithmetic which some accelerators use as their starting point, or to an even lower-level vectorized modular arithmetic level (think of a GPU or TPU for 64-bit integer math modulo custom primes, instead of bfloat16) which some accelerators use as their starting point, or even further down to LLVM, and then to x86 or wasm or whatever.

This will be critical for multiple of our hardware partners, including Niobium, Cornami, and Optalysys, all of whose hardware we are planning to target with HEIR. This path is making steady progress, but I expect it to pick up given that we have both a lot more GPU work maturing, and that I personally should have time in the next 6 months to really dig into that work. That will in turn open a lot of potential for optimizations that involve scheduling, kernel fusion, etc.

GPU and TPU support

We have a bunch of parallel efforts at integrating TPU and GPU backends into HEIR (still as library APIs for now). This includes the CHEDDAR library, Belfort’s GPU library (I’m not sure if their project codename is public), and CROSS/jaxite. We also have heard interest from a variety of other groups working on GPU acceleration.

In my view, GPU/TPU will be crucial to getting HE in production before custom ASICs can be scaled up in datacenters.

New schemes

Researchers are coming out with new schemes every few years, and HEIR is meant to support all of them. Some that are on our radar include

• Gentry-Lee which is optimized for matrix multiplication and has a lot of potential for GPU/TPU synergies.
• Poulpy which streamlines and simplifies a lot of the underlying polynomial math of CKKS, and also makes switching between schemes much simpler. This has already had some progress.
• Gao-Zheng which uses a fancy encoding to enable both arithmetic and logical style operations in one scheme.

Benchmarking

One of the big selling points of HEIR is benchmarking. In particular, I want to be able to benchmark all the different hardware solutions, HE schemes, and optimization techniques so that I can be informed enough to decide what special hardware Google should integrate into its datacenters (when the time comes).

As part of that my colleague Shruthi Gorantala has been hard at work aligning the industry around a benchmarking solution in the form of fhe-benchmarking.org.

We want to add features to HEIR so that it can generate code in the right structure to be submitted to the benchmarking repository. This would make it very easy for people working on HEIR (particularly researchers) to test and showcase the benefits of their optimizations.

Research

A huge part of the draw of HEIR is that researchers can implement their research ideas directly in HEIR, or at least use HEIR as a frontend. So a big part of my job is to help train graduate students on working with HEIR, reviewing their code, etc.

Though HEIR still doesn’t have a proper publication yet, it has been racking up citations, and, to my count, four published papers were based on research done directly in HEIR.

Join us!

If this interests you, or if you want to get some experience in compilers/MLIR or cryptography, HEIR is fully open source, and we have both a weekly office hours and a (recorded) monthly meeting where we review work that happened in the past month and discuss upcoming design work. Everyone is welcome, but we just ask that you show up to one of these in-person meetings (or schedule a private meeting with a maintainer) before submitting a PR, which is part of our policy to reduce AI spam.