© The Author(s) 2025. Published by EDP Sciences and China Science Publishing & Media Ltd.

1. Introduction

Transformer models [1] have attracted much attention for their high performance in practical tasks [2, 3] and have been widely used in online applications [4–6] under Deep Learning as a Service (DLaaS) paradigm [7]. However, these applications can raise privacy concerns, because they require users to reveal their input data to the service provider. These data may contain sensitive information such as chat logs, personal pictures, or bank transactions.

One solution to address the privacy concerns of Transformer models service is Secure Multi-Party Computation (MPC) [8–10], which can keep data and model weights private during inference. Table 1 shows many existing works have proposed various ways to support secure Transformer model inference using MPC, but these approaches are still struggling with one or several of the following challenges:

Challenge-❶: High Inference Cost. Most importantly, there are still performance gaps between current MPC-based Transformer inference frameworks and large models used in real world. Due to the high computational and communication overhead, these systems have been limited to relatively small and simple transformer models (e.g., Bert and GPT2 with a few millions of parameters) [11–17]. Recently, MPC-based frameworks [12, 13] have made considerable improvements, and demonstrated the ability to perform secure inference at the scale of LLaMA-7B. However, there remains considerable overhead: For instance, using BumbleBee [12], the server and the client might need more than 13 minutes to generate 1 token for 8 token prompts using LLaMA-7B. SIGMA [13] is fast in the online phase, but requires expensive preprocessing to generate hundreds of gigabytes of function secret sharing (FSS) keys for each query.

Challenge-❷: Retraining (Fine-tuning) Required. To reduce the cost of non-linear functions, several works [11, 17, 18] suggested approximating GELU, Exp, and Softmax using simpler functions like ReLU and quadratic polynomials. These functions are up to an order of magnitude cheaper in MPC but would introduce utility loss to the Transformer model. As a result, they require an extra step of model retraining/fine-tuning. BOLT [16] also applies quantization-aware fine-tuning [19] to counteract the accuracy loss from small bit-width and scales. Ditto [20] utilizes distillation-based fine-tuning to quantize the Transformer models with small bit-widths for better efficiency. However, retraining/fine-tuning is unfriendly for data-limited participants, and might not achieve satisfactory model performance [21].

Challenge-❸: Incompatible Architectures. Works [11, 22] proposed to modify the architecture of Transformer models to accelerate secure inference, e.g., decompose the embedding procedure, reorganize the linear layers, and replace LayerNorm with BatchNorm. Some other works define the Transformer models from scratch in their customized implementation (e.g., C++) [13–17, 23]. They require considerable manual effort to modify and convert an existing cleartext model into an MPC one. Since the machine learning area is fast evolving, and new models are coming out every week, such requirements might not be satisfactory for agile deployment. It will be much more competitive if the service provider could seamlessly load pre-trained cleartext models.

Challenge-❹: Limited Modalities. Existing works [11–18, 22] primarily focus on simple Transformer models (e.g., Bert) and tasks (e.g., classification). However, practical applications often require handling more complex tasks (e.g., text summarization and translation) and diverse modalities (e.g., audio/image-to-text), where the effectiveness of existing works remains unknown.

To summarize, in the field of MPC Transformer-based serving framework, achieving model performance, efficiency, compatible architectures, and diverse modalities are challenging, and people may ask:

Could pre-trained large Transformer models be securely and efficiently evaluated on real-world tasks with comparable performance as cleartext and compatible architectures with existing Transformer libraries, and without further retraining/fine-tuning?

To address these challenges, we propose our PUMA, a fast and accurate end-to-end MPC-secUre TransforMer-based serving framework that can support LLaMA-7B. Our contributions are as follows:

• Protocols for Non-linear Functions of Transformer. We propose accurate and fast MPC protocols for expensive non-linear functions of Transformer models, including GELU, Softmax, LayerNorm, and Embedding. By designing MPC-friendly yet accurate approximations based on the specialized properties of these functions, we achieve both high accuracy and efficiency. Additionally, our protocols align with the workflows required by standard cleartext Transformer libraries. Some works such as BumbleBee [12] and Ditto [20] also follow this way, but PUMA is the first to lead this trend.

• End-to-End Framework Compatible with Cleartext. Benefiting from the high-accuracy and aligned-workflows of our protocols, we implement all the layers required by the Transformer in MPC, following the workflows of cleartext Transfomer libraries. This allows us to easily load and securely evaluate the pre-trained cleartext Transformer models (e.g., downloaded from Hugging Face). To our best knowledge, PUMA is the first open-sourced MPC solution that supports accurate inference of pre-trained Transformer models without further modifications such as retraining/fine-tuning. We opensource PUMA at https://github.com/AntCPLab/puma_benchmarks.

• Fast and Accurate Secure Transformer Model Inference. We make extensive experiments on 10 Transformer models and 8 datasets with devise modalities, including text classification, generation, summarization, translation, audio-to-text, and image-to-text. Results show PUMA’s model performance is similar to cleartext ones’ and is about 2× faster than CRYPTEN (note CRYPTEN does not achieve comparable model performance as PUMA) and SecretFlow-SPU. PUMA can even evaluate LLaMA-7B in less than 5 minutes to generate one word given 8 token prompts.

Remark. We are the first to propose accurate and efficient approximations for complex activation functions in Transformer models. Following our work, several recent studies [12, 16, 20, 23] have proposed their own approximations or optimized our methods in their respective settings, and have cited our preprint version [25].

Organization. We introduce the background in Section 2. The overview of our architecture is given in Section 3 and the detailed design is in Section 4. We analyze the experiments in Section 6. Related works are summarized in Section 7 and our conclusion is in Section 8.

2. Preliminary and background

2.1. Notations

The main used notations are as follows: P_i represents the i-th computing party, i ∈ {0, 1, 2}. The uppercase bold letter X is used for matrices, and the lowercase bold letter x denotes vectors. x[j] denotes the j-th element of vector x, while lowercase letter x is used for scalar values. ℤ_2^ℓ denotes the discrete ring modulo 2^ℓ, ℝ denotes real numbers. ⟦ ⋅ ⟧ is used for 2-out-of-3 replicated secret sharing [26, 27].

2.2. Transformer model

Transformer models have achieved remarkable success in language understanding [2, 28–30], vision understanding [3, 25, 31], and etc. Two popular variants are Bert (Bidirectional Encoder Representations from Transformers) [28] and GPT (Generative Pre-Trained models) [2]. A Transformer model [1] mainly consists of Embedding, Attention, Feed-Forward Network, and LayerNorm sub-layers. Given a token (e.g., a word) id, the Embedding layer maps it to a hidden vector representation by querying the Embedding table. Below, we have given the details of the other sub-layers:

Attention. Given inputs (Q, K, V), the Attention function is computed as Attention(Q, K, V)=Softmax(Q ⋅ K^T + M)⋅V, where M can be viewed as a bias matrix. Besides, Multi-Head Attention [1] can attend to information from different representation subspaces at different positions in parallel.

Feed-Forward Network (FFN). FFN is applied to each position separately and identically. This consists of two linear transformations with an activation in between. Given input x and parameters {W₁, b₁, W₂, b₂}, FFN can be formalized FFN(x)=W₂Act(W₁x + b₁)+b₂, where Act denotes the activation function, and the common activation functions in Transformer include GELU and SiLU [32], which are much more complex than ReLU. Note that the parameters of linear transformations are different from layer to layer.

LayerNorm. Given vector x ∈ ℝⁿ, LayerNorm is defined as: $\mathsf{LayerNorm}(\mathbf{x})[j]=\gamma ·\frac{\mathbf{x}[j]-\mu }{\sqrt{\sigma }}+\beta ,$ where (γ, β) are trained parameters, mean $\mu =\frac{{\sum }_{j=1}^n\mathbf{x}[j]}{n}$, and standard deviation $\sigma =\sum _{j=1}^n{(\mathbf{x}[j]-\mu )}^2$. Compared to BatchNorm, LayerNorm is significantly more complex and expensive during secure inference. This is because the mean and standard deviation (μ, σ) depend on the test data and cannot be accurately approximated by the training dataset, unlike BatchNorm [33].

2.3. 2-out-of-3 replicated secret sharing

Secret sharing is widely used in MPC-based machine learning, where privacy is ensured by keeping all the intermediate values shared between multiple non-colluding computing parties. Here we choose 2-out-of-3 replicated secret sharing (RSS) [27] for its high efficiency. In RSS, a secret value x ∈ ℤ_2^ℓ is shared by three random values x₀, x₁, x₂ ∈ ℤ_2^ℓ with $x=x_0+x_1+x_2(mod{2}^{\ell })$, and party P_i gets (x_i, x_i + 1) (denoted as ⟦x⟧_i). Without special declaration, we compute in ℤ_2^ℓ and omit $(mod{2}^{\ell })$ for brevity. In the case of ℓ >  1 (e.g., ℓ = 64) which support arithmetic operations (e.g., +, −, and ⋅), we refer to this type as Arithmetic Sharing and use notation ⟦ ⋅ ⟧. Boolean Sharing (⟦ ⋅ ⟧^B) refers to ℓ = 1 where ( + , − ) and ⋅ are respectively replaced by bit-wise ⊕ and ∧.

2.3.1. Addition and multiplication

Let (c₁, c₂, c₃) be public constants, and (⟦x⟧, ⟦y⟧) be two secret-shared values. The secure addition and multiplication procedures are as follows:

Addition. ⟦c₁x + c₂y + c₃⟧ can be computed as (c₁x₀ + c₂y₀ + c₃, c₁x₁ + c₂y₁, c₁x₂ + c₂y₂), where P_i can compute its share locally. When (c₁ = 1, c₂ = 1, c₃ = 0), we get ⟦x + y⟧.

Multiplication. In secure multiplication ℱ_Mul(⟦x⟧, ⟦y⟧), parties follow steps: i) First, P_i computes z_i = x_iy_i + x_i + 1y_i + x_iy_i + 1 locally, ii) Parties then perform re-sharing by letting P_i sends z_i′=α_i + z_i to P_i − 1, where α₀ + α₁ + α₂ = 0 (P_i can generate α_i using pseudorandom generators with negligible overhead as [27]). iii) Finally, {(z₀′,z₁′), (z₁′,z₂′), (z₂′,z₀′)} form the 2-out-of-3 replicated secret shares of ⟦x ⋅ y⟧.

2.3.2. 3-Party functionalities

In addition to addition and multiplication, PUMA invokes several other 3-party functionalities from the state-of-the-art works [27, 34, 35] as follows:

• ⟦z⟧ = ℱ_MulBA(⟦b⟧^B, ⟦x⟧), s.t. z = b ⋅ x

• ⟦z⟧ = ℱ_Square(⟦x⟧), s.t. z = x²

• ⟦z⟧^B = ℱ_Eq(⟦x⟧, ⟦y⟧), s.t. z = 1{x = y}

• ⟦z⟧^B = ℱ_LT(⟦x⟧, ⟦y⟧), s.t. z = 1{x <  y}

• ⟦z⟧ = ℱ_Recip(⟦x⟧), s.t. z = 1/x

• ⟦z⟧ = ℱ_rSqrt(⟦x⟧), s.t. $z=1/\sqrt{x}$

• ⟦z⟧ = ℱ_Max(⟦x⟧), s.t. z = maximum(x)

• ⟦z⟧ = ℱ_Trunc^f(⟦x⟧), z = ⌊x/2^f⌋+E, E ∈ {0, ±1}

To simplify the protocol description and security proofs, we describe PUMA using the hybrid model [36].

2.3.3. Fixed-point representation and truncation

Real numbers should be encoded into fixed-point numbers before being represented in finite rings/fields. To avoid overflow, ${\mathcal{F}}_{\mathsf{Trunc}}^f$ has to be used after each fixed-point multiplication to truncate the least f bits securely [37]. For simpler description, we include ${\mathcal{F}}_{\mathsf{Trunc}}^f$ in ℱ_Mul and ℱ_Square by default and do not explicitly mention it in our protocol designs.

The above operations can be easily extended to vectors and matrices, and we use the same notation for vector and matrix operations for simplicity [27, 38].

3. Overview of PUMA’s architecture

To achieve secure inference of Transformer models, PUMA defines three kinds of roles: model owner, client, and three non-colluding computing parties. The computing parties should be hosted by different cloud vendors, e.g., Amazon AWS, Google Cloud, and Microsoft Azure. As illustrated in Figure 1, the model provider and client send their respective models and inputs to the computing parties (i.e., P₀, P₁, and P₂) in a secret-shared form (steps ➀ and ➁). Then, the computing parties process and load the model into the MPC world seamlessly, execute MPC protocols on top of the secret-shared data (➂), and send the shared output back to the client, who reconstructs the shares and gets the final result (➃). Note that an entity could play multiple roles, e.g., the model could be provided by one of the cloud vendors, here we describe them separately for generality. Since all the data are kept secret-shared during the whole process, and it is reasonable to assume that big vendors will not collude with each other to reconstruct client’s information, data privacy could be guaranteed.

Figure 1. The overview of PUMA’s workflow. ➀ Model provider secret-shares the model among the computing parties. ➁ Client generates the input-ids for the input file, secret-share the input-ids, and sends the shares to computing parties. ➂ The servers process and evaluate the model on input-ids securely using 3-party protocols. ➃ The servers return results’ shares to the client, who performs reconstruction

For the cost evaluation of PUMA, we have the following declarations:

• For step ➀, the model provider can secret-share the model before service, and the same model can be used to provide services many times. Therefore, the amortized costs of model sharing can be negligible;

• For steps ➁ and ➃, we only need little costs between the computing servers and client to communicate secret shares of inputs and results, which are also negligible compared to the costs of 3-party inference.

• With the above analysis in mind, we mainly focus on the communication and running time of the 3-party inference (step ➂).

During the secure 3-party inference process, a key invariant is maintained: For any layer, the computing parties always start with 2-out-of-3 replicated secret shares of the previous layer’s output and the model parameters¹, and end with 2-out-of-3 replicated secret shares of this layer’s output. As the shares do not leak any information to each party, this ensures the layers can be sequentially combined for arbitrary depths to obtain a secure computation scheme for any Transformer models.

Threat Model. Following works [11, 27], PUMA is secure against a semi-honest (a.k.a., honest-but-curious) adversary that corrupts no more than one of three computing parties, such an adversary will follow the protocol specifications, but may try to learn other’s private information during the protocol. Note that PUMA cannot defend against attacks based on inference results, and the mitigation methods (e.g., differential privacy [39]) fall outside the scope of this study.

4. Design of PUMA

In this section, we first present our consideration about loading pre-trained Transformer models seamlessly, and then give optimizations and designs for the functions of Transformer models. Note that linear layers such as matrix multiplication are straightforward in 3-party computation, so we omit them for brevity.

4.1. Considerations of seamless model loading

As shown in the step ➀ of Figure 1, the parties must load and convert the pre-trained Transformer models from cleartext into MPC world for subsequent service. While some works tried to offer the capability to load pre-trained models, they require considerable efforts to re-define the model from scratch [13–16] or modify the models’ architecture [11, 24]. Note that these efforts are required for each model. These present significant challenges in practical deployments, especially if there are frequent new models. In contrast, once set up, PUMA could load new models without any tailored modifications.

Figure 2. Pre-trained Transformer model processing, loading, and functions hijack in PUMA

As illustrated in Figure 2, we present how to process and load a pre-trained Transformer model into PUMA, which is compatible with different mainstream cleartext ML frameworks. Consequently, given a Transformer program defined in JAX, PUMA will implicitly call JAX API and SecretFlow-SPU [40] frontend (i.e., IR and Optimization) to convert the functions of Transformer models into the backend MPC protocols, e.g., the function GELU can be dispatched as addition, multiplication, and tanh. However, to execute the Transformer models defined in public libraries (e.g., Hugging Face transformers [41]) efficiently, we integrate the following improvements and optimizations.

4.1.1. Serialization and I/O

Existing serialization libraries of SecretFlow-SPU, such as Protobuf [42] and FlatBuffers [43], only support data trunks with size up to 2 GB, which can only support small size Transformer models (e.g., Bert-Base) and is not sufficient for large MPC-based Transformer models (e.g., LLaMA-7B). To address this problem, we propose an optimization in terms of the serialization and I/O sub-systems. Concretely, the system could automatically divide and serialize overly large secret-shared structures into smaller chunks when communicating or performing I/O operations.

4.1.2. Hijack complex function

SecretFlow-SPU currently provides support for vanilla JAX models implemented entirely in Python. It achieves this by directly dispatching model functions as a series of fundamental backend protocols. e.g., GeLU is dispatched as defined in jax.nn.gelu. When evaluating Transformer models securely in SecretFlow-SPU, an evident limitation arises from its lack of consideration for the MPC-friendly approximations for complex functions (e.g., GELU and Softmax). Consequently, developers proposing new approximation methods have to implement a new function from scratch, then re-direct all the calls to their new function, which is a task that poses significant complexity, particularly for those developers without enough MPC expertise. To address this limitation, we exploit decorator contextmanager to hijack the original functions. This allows for the seamless integration of our proposed approximated procedures into the end-to-end pipeline. The process unfolds as follows:

• Automatic Replacement. PUMA automatically substitutes the computation of hijacked functions with our proposed approximated procedures. These procedures are meticulously designed to leverage basic operations while ensuring MPC-friendliness and accuracy.

• Dispatch to MPC Primitives. Following the automatic replacement, PUMA proceeds to dispatch the computation of our proposed approximated procedures into the corresponding MPC protocols. This dispatch is crucial for ensuring compatibility with the underlying secure computation framework.

We show how to hijack GELU functions in Figure 2, and our approximated methods are in Section 4.2.

4.1.3. Compatible function support

As outlined in Section 1, just running Transformer models under MPC is insufficient for an agile serving framework. It is essential to ensure the model architectures in MPC are compatible with the cleartext ones. Current works fail to do this, e.g., [11, 24] necessitate the client to generate a one-hot vector using the token id locally. This deviates from a cleartext Transformer workflow where the one-hot vector is generated inside the model. As a result, they have to carefully strip off the one-hot step from the pre-trained models, and add the step to the client side, which is incompatible with the cleartext system and could be an obstacle for deployment. To circumvent these problems, we propose faithful implementations of secure Embedding and LayerNorm for Transformer models inference, which are given in Section 4.3.

4.2. Optimizations of secure GELU and Softmax

GELU and Softmax are much more expensive than the other functions in secure Transformer inference [11, 14, 17]. To reduce their costs while preserving model performance (e.g., do not need fine-tuning/re-training), we propose several approximation and optimizations for GELU and Softmax.

4.2.1. Optimizations of secure GELU

We give our design of approximated piece-wise polynomials for GELU function, and then propose their secure evaluation protocol.

Approximated Polynomials. Most of the current approaches view the GELU function as a composition of smaller functions and try to optimize each piece of them, making them miss the chance of optimizing the secure GELU as a whole. Given the GELU function:

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill \mathsf{GELU}(x)& =\frac{x}{2}·(1+tanh(\sqrt{\frac{2}{\pi }}·(x+0.044715·x^3)))\hfill \\ \hfill & \approx x·\mathsf{s}igmoid(0.071355·x^3+1.595769·x)\hfill \end{array},\end{array}$$(1)

these approaches [14, 44] focus either on designing approximate protocols for function tanh or using existing general MPC protocols of exponentiation and reciprocal for sigmoid.

Figure 3. The Mean Square Error (MSE) for different hyperparameters of approximated GELU and negative exponentiation. 3(a) is for MSE with different T1 in domain [ − 4, 3], and 3(b) is for MSE with t for domain [ − 14, 0]

However, we observe the fact that the GELU function is almost linear on the two sides (i.e., GELU(x)≈0 for x <  −4 and GELU(x)≈x for x >  3). Within the short interval [ − 4, 3] of GELU, we suggest a piece-wise approximation of low-degree polynomials are more efficient and easy-to-implement choice for its secure protocol. Concretely, our piece-wise low-degree polynomials for GELU are shown in equation (2):

$$\begin{array}{c}\hfill \mathsf{A}pp\mathsf{GELU}(x)=\{\begin{array}{cc}0,\hfill & x<T_0\hfill \\ F_0(x),\hfill & T_0\le x<T_1\hfill \\ F_1(x),\hfill & T_1\le x\le T_2\hfill \\ x,\hfill & x>T_2\hfill \end{array},\end{array}$$(2)

where the segments parameters (T₀ = −4, T₁ = −1.95, T₂ = 3). For threshold T₁ and the polynomials F₀ and F₁ as follows: We first attempted to fit the interval [ − 4, 3) using a single low-degree polynomial (degree ≤ 6), but the approximation error remained high. We then adopted a two-polynomial strategy, employing a binary search with a stride 0.01 to identify T₁. Balancing accuracy and polynomial degree, we selected T₁ = −1.95 as shown in Figure 3(a), and we can get sparse polynomials by using library numpy.ployfit² as in equation (3). The above simple polyfit works very well and our mean error is less than 0.003 for ℤ_2⁶⁴ with f = 18 fractional bits. We compare the absolute errors of our and prior approximation methods for GELU functions in Figure 4.

$$\begin{array}{c}\hfill \{\begin{array}{c}{F}_0(x)=-0.011034134030615728x^3-0.11807612951181953x^2\hfill \\ -0.42226581151983866x-0.5054031199708174\hfill \\ F_1(x)=0.0018067462606141187x^6-0.037688200365904236x^4\hfill \\ +0.3603292692789629x^2+0.5x+0.008526321541038084\hfill \end{array},\end{array}$$(3)

Figure 4. The absolute errors of our and prior methods for GELU functions

We focus on specialized approximations for nonlinear functions in Transformer models. For low-degree polynomial approximations towards general functions, please refer to [45].

Secure AppGELU Evaluation. Given AppGELU as equation (3), we need secure segment selection and evaluation of F₀ and F₁ to reach the final protocol. The secure segment selection can be achieved by Less-Than functionality ℱ_LT. For secure evaluation of F₀ and F₁, we exploit ℱ_Square to compute all even-power terms {⟦x²⟧, ⟦x⁴⟧, ⟦x⁶⟧}. Also, we require ℱ_Mul to compute ⟦x³⟧. Finally, we use the ℱ_MulBA to select the result securely. Formally, our secure GELU protocol Π_GELU is constructed in Algorithm 1.

4.2.2. Optimizations of secure Softmax

In the function Attention(Q, K, V)=Softmax(Q ⋅ K^T + M)⋅V, the key challenge is computing function Softmax. For the sake of numerical stability, the Softmax function is computed as

$$\begin{array}{c}\hfill \mathsf{Softmax}(\mathbf{x})[j]=\frac{exp(\mathbf{x}[j]-\overline{x}-ϵ)}{{\sum }_{j}exp(\mathbf{x}[j]-\overline{x}-ϵ)},\end{array}$$(4)

where $\overline{x}$ is the maximum element of the input vector x. For the normal plaintext Softmax, ϵ = 0. For a two-dimensional matrix, we apply equation (4) to each of its row vectors.

Equation (4) indicates that the key challenges of secure Softmax are secure Exp and Division. As a consequence, we propose two optimizations for better efficiency.

Exp Approximation. For the secure evaluation of Exp, we set ϵ in equation (4) to a tiny and positive value, e.g., ϵ = 10⁻⁶, so that the inputs to exponentiation in equation (4) are all negative. We exploit the negative operands for acceleration. Particularly, we compute the exponentiation using the Taylor series [24, 46] with a simple clipping

$$\begin{array}{c}\hfill \mathsf{n}egExp(x)=\{\begin{array}{cc}0,\hfill & x<T_{exp}\hfill \\ {(1+\frac{x}{2^t})}^{2^t},\hfill & x\in [T_{exp},0].\hfill \end{array}\end{array}$$(5)

Indeed, we apply the Less-Than functionality ℱ_LT for the branch x <  T_exp. The division by 2^t can be achieved using ℱ_Trunc^t since the input is already negative. Also, we can compute the power-of-2^t using t-step sequences of Square function ℱ_Square. Suppose that our MPC program uses fixed point precision of 18 bits. Then, we set T_exp = −14 given Exp(−14)< 2⁻¹⁸. With this setting, choosing t = 5 yields an average error of around 10⁻⁵ as shown in Figure 3(b), which provides a practical balance between accuracy and efficiency.

Figure 5. Broadcast optimization in Softmax

Broadcast Optimization. In the Division of Softmax, we denote $v=\sum _j\mathsf{Exp}(\mathbf{x}[j]-\overline{x}-ϵ)$ and numerator $\mathbf{u}[j]=\mathsf{Exp}(\mathbf{x}[j]-\overline{x}-ϵ)$. Our second optimization is to reduce the number of Division, which ultimately saves computation and communication costs. Existing works [35, 47] achieve the Division securely by following the Broadcast-Division paradigm: as shown in the upper part of Figure 5, parties sum up all values of u as v (step ➀), and then Broadcast(v) to the same dimension (e.g., n) of u as step ➁ (duplicate v into n copies), and finally invoke the secure Division (step ➂), which requires much computation and communication costs.

Surprisingly, we observe that denominator v is a scalar while numerator u is a vector, which means the denominator is the same for all elements of the numerator. Inspired by the above observation, we replace the operation Div(u, Broadcast(v)) with $\mathbf{u}·\mathsf{Broadcast}(\frac{1}{v})$ as steps ➁-➃ of the bottom part of Figure 5. By making this replacement, we effectively reduce n divisions to just one reciprocal operation and n multiplications. This optimization is particularly beneficial in the case of the Softmax operation. The $\frac{1}{v}$ in the Softmax operation is still large enough to maintain sufficient accuracy under fixed-point values. As a result, this optimization can significantly reduce computation and communication costs while still providing accurate results. Formally, protocol Π_Softmax is illustrated in Algorithm 2.

4.3. Compatible secure Embedding and LayerNorm

We present our faithful implementations of secure Embedding and LayerNorm as follows.

4.3.1. Workflow of secure embedding

Assuming that token id ∈ [n] and all embedding vectors are denoted by E = (e₀, e₁, …, e_n − 1), the embedding can be formulated as e_id = E[id]. Given (id, E) are in secret-shared fashion, our secure embedding protocol Π_Embed works as follows:

• The computing parties securely compute the one-hot vector ⟦o⟧^B after receiving ⟦id⟧ from the client. Specifically, ⟦o[j]⟧^B = ℱ_Eq(⟦id⟧, j) for j ∈ {0, 1, 2, …, n − 1}.

• The parties compute embedded vector as ⟦e_id⟧ = ℱ_MulBA(⟦o⟧^B, ⟦E⟧), which does not require truncation.

In this way, Π_Embed requires more ℱ_Eq operations than MPCFORMER [11], but we do not require explicit modification of the architectures of cleartext Transformer models.

4.3.2. Construction of secure LayerNorm

Recall that given a vector x of size n, $\mathsf{LayerNorm}(\mathbf{x})[j]=\gamma ·\frac{\mathbf{x}[j]-\mu }{\sqrt{\sigma }}+\beta$, where (γ, β) are trained parameters, $\mu =\frac{{\sum }_{j=1}^n\mathbf{x}[j]}{n}$, and $\sigma =\sum _{j=1}^n{(\mathbf{x}[j]-\mu )}^2$. In MPC, the key challenge is the evaluation of the divide-square-root $\frac{\mathbf{x}[j]-\mu }{\sqrt{\sigma }}$. To securely evaluate this formula, CRYPTEN sequentially executes the MPC protocols of square-root, reciprocal, and multiplication.

However, we observe that $\frac{\mathbf{x}[j]-\mu }{\sqrt{\sigma }}$ is equal to (x[j]−μ)⋅σ^−1/2. On the MPC side, the costs of computing the inverse-square-root σ^−1/2 are similar to that of the square-root operation [34]. Besides, inspired by the second optimization about Broadcast of Section 4.2.2, we can first compute σ^−1/2 and then Broadcast(σ^−1/2) to support fast and secure LayerNorm(x). Our protocol Π_LayerNorm is illustrated in Algorithm 3.

5. Security analysis

We capture the security of protocols Π_GELU, Π_Softmax, Π_Embed, and Π_LayerNorm, in Theorem 2–5 under ℱ-hybrid model against semi-honest adversary in 3-party honest-majority setting. And then, we give a proof analysis.

6. Experimental evaluations

We would like to study the performance and efficiency of PUMA in the following four aspects:

• Q1: Can PUMA support various Transformer models with diverse modalities? What is the difference in model performance between PUMA and cleartext? (Section 6.2)

• Q2: What are the advantages of communication and running time of our protocols over existing works? (Section 6.3)

• Q3: What are the improvements in end-to-end efficiency achieved by PUMA compared to existing works? (Section 6.4)

• Q4: Can PUMA be generalized to Large Language Model? How about the model performance and efficiency? (Section 6.5)

In this section, we provide PUMA’s implementation details and setup, with the answers to all questions.

6.1. Experimental setup

6.1.1. Testbed environments

We implement PUMA on top of SecretFlow-SPU [40] in C++ and Python version 3.10. We utilize Flax library [48] to load and run various models from Hugging Face Transformer library [41]. Experiments are run on 3 cloud servers with 128 vCPUs and 512GB RAM each. The CPU model is Intel Xeon(Ice Lake) Platinum 8369B CPU @ 2.70GHz. The Operating System is Ubuntu 20.04.6 LTS with Linux kernel 5.4.0-144-generic. In LAN, bandwidth is about 5 Gbps and round trip time (rtt) is 1.5 ms. In WAN, bandwidth is about 400 Mbps and rtt is 10 ms.

6.1.2. Models, datasets, and metrics

We select 10 Transformer models: Bert-Base/Large and Roberta-Base [28], GPT2-Base/Medium/Large [2], T5-Small [49], Whisper-tiny [50], VisionEncoderDecoder [51], and LLaMA-7B [30]. The architectures of the models are illustrated in the supplementary materials. The datasets, tasks, and metrics are as follows:

• Text Classification aims to classify the text into different categories. We select Bert-Base, Roberta-Base, and Bert-Large for three text classification tasks over the datasets of Corpus of Linguistic Acceptability (CoLA), Recognizing Textual Entailment (RTE), Stanford Question Answering Dataset (QNLI) from GLUE benchmarks [52]. Matthews correlation (range is [ − 1, 1], higher is better) [53] is reported for CoLA. Accuracy (with a range [0, 1], higher is better) is for others.

• Text Generation generates the next tokens from a given sentence. We utilize three GPT2-based models: GPT2-Base, Medium, and Large, for the text generation task. We measure their perplexity [54] on Wikitext-103 V1 [55]. The range of this metric is [0, +∞), a lower score is better.

• Text Summarization can summarize the given long text, and output short summarization. We select T5-Small on dataset CNN-Daily-Mail [56] for this task. And we report ROUGE scores [57] to show model performance. The ROUGE values are in the range of 0–1 and higher score indicates better performance.

• Text Translation translates a given text from a given language to another. We also select T5-Small for this task and evaluate it on dataset WMT16 (de-en split) [58], and measure ROUGE scores as well.

• Audio-to-Text models receive an audio file, process it, and decode predicted IDs back into text. We evaluate Whisper-tiny on dataset librispeech-asr [59] and report WER score [60], where the lower the value, the better model performance, and WER = 0 is a perfect score.

• Image-to-Text models can generate the descriptions for a given image, e.g., image captioning. We evaluate VisionEncoderDecoder on dataset COCO-2017 [61] for image captioning and report the BLEU score [62]. This metric can take on values from [0, 1]. Higher scores are better, with 0 indicating no matches, and 1 indicating a perfect match.

We randomly sample 100 instances from the test/validation split of each dataset to assess the model performance. For the evaluation of LLaMA-7B, we present the details in Section 6.5. To validate PUMA achieves comparable model performance to cleartext, our emphasis lies in evaluating the gap between PUMA and cleartext using pre-trained models, rather than training for the best model performance.

6.1.3. Baselines

We compare PUMA to the most similar prior work CRYPTEN [24], MPCFORMER [11], and SecretFlow-SPU, to show our improvements thoroughly: i) We re-run CRYPTEN and MPCFORMER in our environment for fair comparisons. We achieve the CRYPTEN-based secure inference of Transformer models on top of MPCFORMER by replacing the Quad approximations of GELU and Softmax with the in-built basic protocols (i.e., tanh and Exp) of CRYPTEN. ii) As for SPU, we integrate our serialization and I/O optimization, Π_Embed, and Π_LayerNorm into SPU to make it can evaluate Transformer models securely. We use the default methods for GELU and Softmax in SPU to show our efficiency improvements.

We also have the following considerations: As MPCFORMER neither supports loading pre-trained Transformer models nor implements LayerNorm faithfully³, we cannot achieve meaningful secure inference results using their framework. Therefore, we compare our model performance to cleartext (floating-point) to show our model performance guarantee.

6.1.4. Concrete parameters

We set ring bit-width ℓ = 64 and fractional bit-width f = 18 for Bert-Base, Roberta-Base, Bert-Large, GPT2-Base, GPT2-Median, GPT2-Large, Whisper, and LLaMA-7B models; and (ℓ=128, f = 26) for T5-Small and VisionEncoderDecoder models. For the approximation of GELU, we set (T₀ = −4, T₁ = −1.95, T₂ = 3). For the approximation of negExp, we set T_exp = −14 and t = 5.

6.2. Model performance

We compare PUMA’s model performance (i.e., accuracy) to that of cleartext (floating-point) in Tables 2–4.

Table 2 shows the results of Bert-based and GPT2-based models, we observe that the model performance achieved by PUMA is comparable to that of the cleartext world. Specifically, the performance difference of Bert-based models does not exceed 0.011 over all datasets. For the GPT2-based models, the experimental perplexities are at the same level and the differences do not exceed 0.02 over all models. For Text Summarization, Translation, Audio-to-Text, and Image-to-Text tasks, we set the maximum model output length to 32 tokens (instead of 1) and report performance over all generated tokens. In Table 3, we evaluate T5-Small for text summarization and translation. For text summarization, we can see that our loss of recall, precision, and F1 score are respective no more than 0.01, 0.016, and 0.013. For text translation, our loss of recall, precision, and F1 score are all no more than 0.007. In Table 4, we report the scores of Whisper and VisionEncoderDecoder to show PUMA’s ability to process multi-modal documents (Audio/Image to Text). For Whisper, PUMA’s WER score degradation is around 0.05. As for VisionEncoderDecoder, our loss for BLEU is no more than 0.04 and we achieve comparable precision to cleartext world.

The above metrics’ (i.e., accuracy) advantages experimentally validate that PUMA attains comparable levels of model performance compared to the cleartext inference. Importantly, all our experiments were conducted using the proposed 3PC protocols rather than through cleartext simulation. Also, PUMA achieves such results in a simple “download-and-run” way, without any fine-tuning/re-training.

6.3. Microbenchmarks

To demonstrate the improvements introduced by protocols Π_GELU and Π_Softmax in inference, we evaluate Bert-Base and GPT2-Base using SPU, SPU with Π_GELU but without Π_Softmax (SPU+Π_GELU), and PUMA. The experimental results are depicted in Figure 6: i) Regarding the communication costs, it is evident that Π_Softmax exhibits more significant improvements compared to Π_GELU. ii) Concerning the running time, we observe a similar trend, although the differences in improvements are not as pronounced as those in Figure 6(a). This discrepancy arises because computational tasks (e.g., large matrix multiplication) also contribute to the overall time. Consequently, the reduction in running time does not precisely mirror that of communication costs.

Figure 6. Improvements of communication and running time by protocols ΠGELU and ΠSoftmax. We select Bert-Base and GPT2-Base for evaluations. Figure 6(a) is for communication, 6(b) and 6(c) are for running time in LAN and WAN. SPU+ΠGELU indicates we run SPU with our ΠGELU but without ΠSoftmax (with SPU’s default secure Softmax)

Next, we evaluate the detailed efficiency of Π_GELU and Π_Softmax and report the communication costs and running time in Table 5: i) Compared to SPU, our protocols are more communication efficient and faster for both GELU and Softmax. For GELU, we reduce the communication costs by 20% and running time by 24% on average; For Softmax, we reduce the communication costs by 76% and running time by 66% on average. Also, it is obvious that the improvements of Π_Softmax are more significant than that of Π_GELU. This is consistent with the analysis of Figure 6. ii) Compared to CRYPTEN, we reduce the running time by 27% for GELU and 72% for Softmax on average. We reduce the communication costs by 44% for Softmax. However, our Π_GELU requires more communication costs than CRYPTEN. This is because CRYPTEN utilizes the function HardTanh⁴ to approximate GELU, which results in much large approximated errors than SPU and Π_GELU. iii) Although MPCFORMER requires much less communication and running time than other methods for both functions (because they use Quad-based approximations), it introduces more absolute approximated errors than ours. We provide the costs of Π_LayerNorm and Π_Embed in Table 5: Π_LayerNorm requires fewer costs than Π_GELU and Π_Softmax.

6.4. End-to-End costs on transformer models

We compare our communication and running time to baselines in Figures 7 and 8. For Bert-Base/Large and Roberta-Base, we select Bert-Base as they share a similar architecture. We use GPT2-Base to represent GPT2-Base/Medium/Large. For Bert/GPT2-Base, we compare PUMA to all baselines. For other models, we compare PUMA to SPU as CRYPTEN and MPCFORMER do not support their secure evaluation now.

Figure 7. End-to-end communication costs of Transformer models. #IN denotes the number of input tokens. MPCFORMER uses Quad approximation on top of CRYPTEN. For Whisper, the audio’s features vector is of size [1, 80, 300]. For VisionE.D., the input image is of size [1, 3, 224, 224]. For T5-Small with text translation, we use the default #OUT determined by models and omit it. Bert-Base outputs classification label and we omit its #OUT. For others, #OUT denotes the number of generated tokens. (a) Bert-Base, (b) GPT2-Base, (c) T5-Small, Summ., (d) T5-Small, Trans, (e) Whisper & VisionE.D.

Figure 8. Running time of Transformer models. Figures 8(a)–8(e) are for LAN and 8(f)–8(j) are for WAN. (a) Bert-Base, (b) GPT2-Base, (c) T5-Small, Summ., (d) T5-Small, Trans., (e) VisionEncoderDecoder, (f) Bert-Base, (g) GPT2-Base, (h) T5-Small, Summ., (i) T5-Small, Trans., (j) Whisper and VisionE.D.

From Figure 7, we can see that: i) Compared to CRYPTEN, we reduce the communication costs by 17 − 48%. At the same time, the improvements fraction decreases with the increases of #IN. This is because our protocol Π_Embed needs to compute the one-hot vector with ℱ_Eq securely, while we run CRYPTEN-based secure Transformer inference based on MPCFORMER, which computes the one-hot vector in cleartext⁵. ii) PUMA is more expensive than MPCFORMER. This is not unexpected since MPCFORMER use Quad-based approximation for GELU and Softmax, which has a high approximation error (c.f. Section 6.3) and require complicated retraining/fine-tuning for reasonable model performance. iii) Compared to SPU, PUMA reduces the communication by 29 − 69%. These improvements mainly come from our optimizations on GELU and Softmax.

Figure 8 shows the running time in LAN and WAN settings: i) Compared to CRYPTEN, we reduce the running time by 33 − 53% in LAN and roughly 17% in WAN. ii) Similarly, PUMA requires more running time than MPCFORMER since MPCFORMER’s approximation methods are more MPC-friendly (but introduce more errors). iii) Compared to SPU, we reduce 36% and 26% of the running time on average in respective LAN and WAN. From the secure evaluation of T5-Summarization and Whisper, we find that #IN has a more significant impact on the running time than #OUT. With the increase of #IN and #OUT, our improvements are likely more significant because our optimizations usually have more benefits when processing large-scale evaluations.

6.5. Evaluating LLaMA-7B in five minutes

Until now, our protocols are already complete for evaluating any Transformer-based models including LLaMA-7B. Unlike GPT-2 and Bert, LLaMA-7B uses SiLU instead of GELU, we can approximate SiLU using similar piece-wise low-degree polynomials with different coefficients:

$$\begin{array}{c}\hfill \mathsf{A}pp\mathsf{SiLU}(x)=\{\begin{array}{cc}0,\hfill & x<-8\hfill \\ G_0(x),\hfill & -8\le x<-4\hfill \\ G_1(x),\hfill & -4\le x\le 4\hfill \\ x,\hfill & x>4\hfill \end{array},\end{array}$$(6)

where the polynomials G₀(x) and G₁(x) are as follows:

$$\begin{array}{c}\hfill \{\begin{array}{c}{G}_0(x)=-0.3067541139982155x^2-0.0819767021525476x-0.0055465625580307,\hfill \\ G_1(x)=0.0085064025895951x^6+0.5x^4+0.2281430841728270x^2\hfill \\ -0.011113046708173x+0.0002743776353465.\hfill \end{array}\end{array}$$(7)

We evaluated the large language model LLaMA-7B using PUMA under 3 Alibaba Cloud ecs.r7.32xlarge servers, each having 128 vCPUs and 1 TB RAM, with 20 Gbps bandwidth, 0.15 ms round-trip-time. We randomly sample 50 instances from the validation split of dataset Wikitext-103 V1 with #IN=64, evaluate LLaMA-7B on them for text generation (1 token), and measure the perplexity. In plaintext, our experimental perplexity is 14.593. And the perplexity of PUMA is 14.528. The difference is very small and demonstrates that PUMA can guarantee the model performance of the large language model to some extent. PUMA can support secure inference of LLaMA-7B with reasonable costs. As shown in Table 6, given an input sentence of 8 tokens, PUMA can output 1 token in around 200 seconds with communication costs of 1.794 GB. More results are in the supplemental materials.

Moreover, in Figure 9, we show the output tokens of LLaMA-7B (with fixed randomness) given the same prompt. It can be seen that PUMA outputs the same tokens as LLaMA-7B in cleartext for generating over 20 tokens.

Figure 9. Example of LLaMA-7B in plaintext and PUMA

6.6. Comparison with other frameworks

Comparison with 3PC frameworks. A concurrent work [13] proposes a very efficient 3PC system SIGMA that is also able to evaluate Transformer models with billions of parameters. PUMA is still more efficient than theirs considering the end-to-end evaluation. For example, SIGMA requires the third dealer to distribute 16.69 GB FSS keys to each party for the Bert-Base model processing a sentence of 128 tokens [13]. Assuming our LAN setting (5 Gbps bandwidth), this would add 16.69 × 8 × 2/5 = 53.4 seconds to the total running time. With our WAN setting (400 Mbps bandwidth), this would add 684 seconds to the total running time. The time is almost 2× expensive than ours (c.f., Figure 8(a) and 12(a)). Also, the size of FSS keys are almost 5× of our communication (c.f., Figure 7(a)). Consequently, PUMA is more efficient in end-to-end communication and running time. Ditto [20] shares a similar design with PUMA and proposes several bit-width conversion protocols for quantized Transformer models. Ditto requires distillation-based fine-tuning to guarantee model performance, whereas PUMA does not. What’s more, Ditto’s accuracy is lower than PUMA (Table 1, [20]).

Comparison with 2PC frameworks. Works like Iron [14], BumbleBee [12], CipherGPT [15], and BOLT [16] support 2PC secure inference of Transformer models. They have a different threat model as 3PC ones and it might be not fair to directly PUMA’s efficiency to theirs, we discuss them here for completeness. As shown in Table V of [15], CipherGPT requires a communication cost of more than 14 GB to generate one token given 256 tokens for GPT2-Base. While we need to transfer 8.831 GB messages. According to the evaluation of BumbleBee (Table V, [12]), our PUMA requires much less communication and running time than IRON and BOLT: i) Compared to IRON, we reduce the communication by around 7× and running time by more than 15×. ii) Compared to BOLT, we reduce the communication by around 5× and running time by around 4×. While we have a slightly higher total communication cost than BumbleBee due to the redundancy of replicated secret sharing, PUMA is still up to 3× faster because BumbleBee heavily relies on Homomorphic Encryption [63] for matrix multiplication.

7. Related work

Secure Multiparty Computation (MPC) [9, 10] enables distrusted parties to jointly compute a function while keeping their inputs private, and secure deep learning inference using MPC has gained much attention due to its high privacy protection. These works operate in a variety of models and architectures, including two-party setting [64–70], three-party setting [24, 27, 38, 46, 71–74], four-party setting [75, 76], and multi-party setting [35, 77]. However, most of these approaches only consider secure inference of convolutional/deep neural networks.

Several frameworks discussed above are capable of evaluating Transformer models securely, e.g., CRYPTEN [24], SIRNN [69], MP-SPDZ [35], and SecretFlow-SPU [40]. For example, Hao et al. propose IRON [14], which integrates SIRNN’s OT-based protocols for the non-linear functions and Cheetah’s secure matrix multiplication [70] to support 2-party secure Transformer inference. In BOLT [16], Pang et al. improves the secure inference efficiency over IRON by combining more efficient 2-party protocols with ML optimizations. Some recent works [12, 15] also propose accurate and fast approximation methods to complex functions and several optimizations to secure matrix multiplication of 2-party Transformer inference. Zhang et al. propose NEXUS, the first non-interactive secure Transformer inference framework based on fully homomorphic encryption [63]. Gupta et al. propose a function secret sharing (FSS)-based SIGMA [13] in the dealer-based 3-party setting, where the trusted dealer generates correlated randomness for the other two computing parties to evaluate the FSS-based protocols. Also, SIGMA exploits GPU for better computational efficiency. Li et al. propose MPCFORMER [11] based on CRYPTEN in the dealer-based 3-party setting with simpler and MPC-friendly approximations for the complex functions. Therefore, it requires knowledge distillation and re-training/fine-tuning to achieve reasonable model performance. Similarly, works [16–18, 22, 78] have also exploited re-training/fine-tuning to improve the efficiency of MPC-based secure inference solutions for Transformer models. [79, 80] exploit activation distributions to accelerate computation, but the distribution generality is empirical. We propose PUMA independent of them; the efficiency and accuracy of our methods are comparable to these methods.

The existing literature primarily concentrates on basic Transformer-based models (e.g., Bert and GPT2) and straightforward tasks like text classification. However, real-world Transformer-based serving frameworks encompass a broad spectrum of tasks, often involving intricate Transformer models and diverse file types [4, 5]. Moreover, real-world applications frequently necessitate the processing of multi-modal data, such as audio and image data. It remains unclear how to effectively apply these existing frameworks to handle diverse modality data in real-world scenarios. To the best of our knowledge, our PUMA stands out as the first experimentally verified solution capable of extending support to diverse multi-modal data/tasks and incorporating complex Transformer-based models.

8. Conclusion

We propose an efficient MPC framework PUMA for providing Transformer-based services. We present how to load pre-trained Transformer models into the MPC domain seamlessly and accurately approximated polynomials for complex functions of the Transformer models. Although the inference cost is still quite high, we successfully made it one step closer to solving privacy concerns in Transformer-based services. By combining PUMA with quantization methods and hardware accelerations such as GPUs, secure inference of large Transformer models in seconds are no longer impossible.

Acknowledgments

No Acknowledgments.

Funding

No fundings are related to this article.

Conflicts of interest

The authors declare no conflicts of interest.

Data availability statement

The data used in this study are publicly available from datasets of GLUE benchmarks [52], Wikitext-103 V1 [55], CNN-Daily-Mail [56], WMT-16 (de-en split) [58], librispeech-asr [59], COCO-2017 [61].

Author contribution statement

Ye Dong contributed to the system design, implementation, experiments, and writing of the manuscript. Wen-Jie Lu contributed to the protocol design and evaluation. Yancheng Zheng supported the development of experimental evaluation and data analysis. Haoqi Wu contributed to the Transformer models processing and implementation. Derun Zhao assisted with prototype implementation. Jin Tan participated in the prototype implementation and performance optimization. Zhicong. Huang contributed to protocol and algorithm optimizations. Cheng Hong supervised the project and guided system design and evaluation. Tao Wei, Wenguang Chen, and Jianying Zhou supervised the research and contributed to the high-level idea.

References

Ye Dong received his Ph.D. degree from the School of Cyber Security, University of Chinese Academy of Sciences in 2023. He is currently a Postdoctoral Research Fellow at the National University of Singapore, Singapore. His research interests include practical secure computation and privacy-preserving machine learning.

Wen-Jie Lu received his Ph.D. degree from the University of Tsukuba, Japan. He is currently a Research Scientist at TikTok, China. His research interests include privacy-preserving machine learning and applied homomorphic encryption.

Yancheng Zheng received his M.S. degree from Rutgers, The State University of New Jersey, USA. He is currently serving as Senior Architect at NVIDIA, China. His research interests include programming languages, compiler techniques, and computer architecture design and optimization.

Haoqi Wu received his M.S. degree from Fudan University. He is currently an algorithm engineer at Ant Group, China. His research interests include privacy-preserving machine learning and AI security.

Derun Zhao is a Staff Engineer at Ant Group, China. His research interests include privacy-preserving machine learning, secure computation, and program optimization.

Jin Tan is currently a Staff Engineer at Ant Group, China. His research interests include privacy-preserving machine learning, secure computation, and compiler optimization.

Zhicong Huang received his Ph.D. degree in computer science from École Polytechnique Fédérale de Lausanne, Switzerland. He is currently a Research Scientist at Ant Group, China. His research interests include security and privacy, applied cryptography, and machine learning.

Cheng Hong received his Ph.D. degree from the University of Chinese Academy of Sciences in 2012. He is currently the Director of Cryptography and Privacy Research at Ant Group, China. His research interests include information security and applied cryptography.

Tao Wei received his B.S. and Ph.D. degrees from Peking University, China, in 1997 and 2007, respectively. He is currently Vice President of Ant Group, China. Prior to joining Ant, he was Head of the Baidu X-Lab. His research interests include software security, web security, mobile security, and data security.

Wen-Guang Chen is a Professor in the Department of Computer Science and Technology, Tsinghua University, Beijing, and Vice President of Ant Group, China. He received the B.S. and Ph.D. degrees in computer science from Tsinghua University in 1995 and 2000, respectively. His research interests include parallel and distributed computing.

Jianying Zhou is a Professor and Center Director for iTrust at the Singapore University of Technology and Design (SUTD), Singapore. He received his Ph.D. degree in Information Security from Royal Holloway, University of London. His research interests include applied cryptography, network security, cyber-physical system security, mobile and wireless security.

All Tables

All Figures

Figure 1. The overview of PUMA’s workflow. ➀ Model provider secret-shares the model among the computing parties. ➁ Client generates the input-ids for the input file, secret-share the input-ids, and sends the shares to computing parties. ➂ The servers process and evaluate the model on input-ids securely using 3-party protocols. ➃ The servers return results’ shares to the client, who performs reconstruction

In the text

	Figure 2. Pre-trained Transformer model processing, loading, and functions hijack in PUMA
In the text

	Figure 3. The Mean Square Error (MSE) for different hyperparameters of approximated GELU and negative exponentiation. 3(a) is for MSE with different T1 in domain [ − 4, 3], and 3(b) is for MSE with t for domain [ − 14, 0]
In the text

	Figure 4. The absolute errors of our and prior methods for GELU functions
In the text

	Figure 5. Broadcast optimization in Softmax
In the text

	Figure 6. Improvements of communication and running time by protocols ΠGELU and ΠSoftmax. We select Bert-Base and GPT2-Base for evaluations. Figure 6(a) is for communication, 6(b) and 6(c) are for running time in LAN and WAN. SPU+ΠGELU indicates we run SPU with our ΠGELU but without ΠSoftmax (with SPU’s default secure Softmax)
In the text

Figure 7. End-to-end communication costs of Transformer models. #IN denotes the number of input tokens. MPCFORMER uses Quad approximation on top of CRYPTEN. For Whisper, the audio’s features vector is of size [1, 80, 300]. For VisionE.D., the input image is of size [1, 3, 224, 224]. For T5-Small with text translation, we use the default #OUT determined by models and omit it. Bert-Base outputs classification label and we omit its #OUT. For others, #OUT denotes the number of generated tokens. (a) Bert-Base, (b) GPT2-Base, (c) T5-Small, Summ., (d) T5-Small, Trans, (e) Whisper & VisionE.D.

In the text

	Figure 8. Running time of Transformer models. Figures 8(a)–8(e) are for LAN and 8(f)–8(j) are for WAN. (a) Bert-Base, (b) GPT2-Base, (c) T5-Small, Summ., (d) T5-Small, Trans., (e) VisionEncoderDecoder, (f) Bert-Base, (g) GPT2-Base, (h) T5-Small, Summ., (i) T5-Small, Trans., (j) Whisper and VisionE.D.
In the text

	Figure 9. Example of LLaMA-7B in plaintext and PUMA
In the text