LoRA: Low-Rank Adaptation Proposed in a Research Paper

The paper proposes LoRA, which freezes pre-trained model weights and injects trainable rank decomposition matrices into each Transformer layer.

Verified

Definition

The LoRA paper proposes Low-Rank Adaptation, or LoRA, which freezes the pre-trained model weights and injects trainable rank decomposition matrices into each layer of the Transformer architecture, greatly reducing the number of trainable parameters for downstream tasks. The paper's introduction says that adapting a pre-trained model is usually done via fine-tuning, which updates all the parameters of the pre-trained model. The paper says the major downside of fine-tuning is that the new model contains as many parameters as the original model.

The paper's abstract says that as larger models are pre-trained, full fine-tuning, which retrains all model parameters, becomes less feasible. The paper says its authors hypothesize that the change in weights during model adaptation also has a low intrinsic rank, which led to the LoRA approach.

At a glance

The paper says its proposal is agnostic to training objective and focuses on language modeling as the motivating use case. The paper's problem statement considers adapting a pre-trained model to downstream conditional text generation tasks, such as summarization, machine reading comprehension (MRC), and natural language to SQL (NL2SQL).

  • The paper says LoRA adds trainable pairs of rank decomposition matrices in parallel to existing weight matrices.
  • The paper says LoRA trains some dense layers indirectly by optimizing rank decomposition matrices of the layers' change, while keeping the pre-trained weights frozen.
  • The paper says that when deployed, the trainable matrices can be merged with the frozen weights, introducing no inference latency compared to a fully fine-tuned model.
  • The paper says that once deployed, tasks can be switched at a much lower cost by only swapping the LoRA weights as opposed to all the parameters.
  • The paper says LoRA allows quick task-switching when deployed as a service by sharing the vast majority of the model parameters.
  • The paper says LoRA is orthogonal to many prior methods and can be combined with many of them, such as prefix-tuning.

The source does not state the order of these points, so they are listed without arrows.

How it works

The paper notes that a neural network contains many dense layers which perform matrix multiplication, and that the weight matrices in these layers typically have full rank. The paper hypothesizes that the change in weights during model adaptation has a low intrinsic rank, which is the starting point of the LoRA approach. The paper notes that in the Transformer architecture there are four weight matrices in the self-attention module and two in the MLP module.

The paper limits its study to adapting only the attention weights for downstream tasks and freezes the MLP modules, so they are not trained in downstream tasks, both for simplicity and parameter-efficiency. The paper says it is possible to not merge the weights and instead dynamically choose the LoRA modules to use for samples in a batch, for scenarios where latency is not critical. The paper says adapter layers are external modules added to a pre-trained model in a sequential manner, whereas LoRA can be seen as external modules added in a parallel manner.

The paper describes adapter tuning as proposed in Houlsby et al. (2019) as inserting adapter layers between the self-attention module (and the MLP module) and the subsequent residual connection. The paper reports that prefix tuning is difficult to optimize and that its performance changes non-monotonically in trainable parameters. The paper says that reserving a part of the sequence length for adaptation necessarily reduces the sequence length available to process a downstream task.

Example

Using GPT-3 175B as an example, the paper says that deploying independent instances of fine-tuned models, each with 175B parameters, is prohibitively expensive. The paper's abstract reports that, compared to GPT-3 175B fine-tuned with Adam, LoRA can reduce the number of trainable parameters by 10,000 times and the GPU memory requirement by 3 times; this is the paper's report for that model and that fine-tuning baseline, not a figure for every LoRA configuration. The paper also reports that on GPT-3 175B it reduces the VRAM consumption during training from 1.2TB to 350GB.

In the configuration of r=4 with only the query and value projection matrices adapted, the paper reports that the checkpoint size is reduced by roughly 10,000 times (from 350GB to 35MB), and says the 350GB model is still needed during deployment. Using GPT-3 175B as an example, the paper shows that a very low rank, where r can be one or two, suffices even when the full rank d is as high as 12,288. In the paper's GPT-3 175B experiments on WikiSQL and MultiNLI, a rank of one suffices when both Wq and Wv are adapted, while training Wq alone needs a larger r.

The paper also says it does not expect a small r to work for every task or dataset. The paper says its experiments cover tasks from natural language understanding (NLU) to generation (NLG), and that it evaluates RoBERTa and DeBERTa on the GLUE benchmark. The paper also provides an empirical investigation into rank-deficiency in language model adaptation, which it says sheds light on the efficacy of LoRA.

Sources

This page is written from the official sources below. Open them to check the original text.

Verified ·

This page is written by AI from the official sources above and checked automatically. This site has not tested it. Found a mistake? Tell us.