Fine-Tuning Large Language Models For Numerical Stock Price Forecasting: A Parameter-Efficient Approach

Abstract

Stock price forecasting is not a new problem because the financial market is volatile and noisy[1][2]. Conventional econometric models (such as ARIMA, GARCH) and neural networks (such as LSTM, GRU) can frequently fail to respond to the rapid dynamics of the market or have many parameters. Recent developments indicate that big language models (LLMs) can be re-used on time-series problems, which makes use of their high-contextual modelling capabilities[3][4]. Nevertheless, the application of LLMs to numeric forecasting is not a simple task because of the differences in the data between text and time-series. In this work, we propose a parameter-efficient fine-tuning framework that adapts an open-source LLM (LLaMA-3.2, 1B parameters) for stock price forecasting. Our method uses Low-Rank Adaptation (LoRA) modules to inject financial patterns into the LLM without full retraining[5][3]. We train on historical daily closing prices (e.g. AAPL, GOOGL) and evaluate against ARIMA and LSTM baselines. Results show our LoRA-tuned LLM outperforms these baselines with significantly lower forecasting error. For instance, across test sets our fine-tuned LLaMA model achieves ~12–15% lower mean absolute error than a comparable LSTM[6][7]. The LLM’s understanding of long-range dependencies and new training on financial data enable this gain. We provide a detailed analysis of training efficiency, model convergence, and ablation studies. Our contributions include (i) demonstrating that LLMs, when cheaply adapted, can yield state-of-the-art results for stock forecasting, (ii) designing a numeric-to-text prompt strategy for continuous financial data, and (iii) benchmarking performance across multiple datasets with clear improvements. This work opens a path for LLM-based foundation models in quantitative finance, combining deep contextual knowledge with parameter-efficient training techniques[3][7].