2510.0002 Enhancing Small Language Models with Gradient Noise Injection v1

🎯 ICAIS2025 Submission
Back to Paper List

Paper Content

📄 Open in New Tab

AI Review from DeepReviewer

AI Review available after:
--d --h --m --s

📋 AI Review from DeepReviewer will be automatically processed

📋 Summary

This paper introduces a method for enhancing the training of small language models by injecting noise into the gradients during the backward pass. The core idea is to add Gaussian noise to the gradients, with the noise level decreasing over time according to a specific schedule. The authors claim that this approach regularizes the training process, improves robustness, and enhances generalization. They evaluate their method on three text datasets: shakespeare_char, enwik8, and text8, reporting training and validation loss results. The experimental results show that models trained with gradient noise injection achieve lower training and validation losses compared to baseline models. However, the paper's claims of novelty and the robustness of the evaluation are questionable. The method, while straightforward, lacks a clear differentiation from existing techniques, and the evaluation is limited in scope and lacks statistical rigor. These issues undermine the paper's overall significance and the reliability of its findings.

✅ Strengths

The paper is self-contained and mostly easy to follow, which is a positive aspect. The authors clearly describe their objective, which is to improve the training of small language models, and they provide a straightforward method to achieve this. The core idea of injecting Gaussian noise into the gradients during the backward pass is simple and easy to implement, making it accessible to other researchers and practitioners. The authors also present clear experimental results, demonstrating the effectiveness of their approach in reducing training and validation losses on the three datasets used. The method's simplicity and the clarity of the results are commendable, as they provide a solid foundation for further exploration and potential improvements. Additionally, the paper's focus on small language models is relevant, given the computational constraints often associated with these models. The authors' choice to validate their method on shakespeare_char, enwik8, and text8 is reasonable for an initial evaluation, as these datasets are commonly used in the literature.

❌ Weaknesses

Despite the paper's clear presentation and straightforward method, several critical weaknesses significantly impact its overall quality and the reliability of its findings. First, the novelty of the proposed method is limited. Adding noise to gradients during training is a well-established technique that has been explored in numerous papers since the 1980s, including the work by Jacobs et al. (1991). The authors fail to demonstrate any significant novelty in their approach, particularly in the specific noise schedule they use. The exponential decay schedule, while described, is not justified as being superior to other schedules, and the paper lacks a comprehensive comparison with existing methods. This omission makes it difficult to assess the unique contribution of the proposed method and undermines the claim of novelty (Confidence Level: High). Second, the evaluation is inadequate. The authors evaluate their method on only three small text datasets, which is insufficient to demonstrate the general effectiveness of their approach. The reported results lack statistical significance, as the authors do not provide error bars or statistical tests. The improvements in training and validation loss are marginal, and it is unclear whether these improvements are meaningful. For example, the differences in loss values in Table 1 are relatively small, such as 0.8132 vs 0.8083 for training loss on shakespeare_char, and without statistical testing, it is impossible to determine if these differences are due to the method or random variation (Confidence Level: High). Third, the paper is missing relevant related work. The authors fail to cite and discuss important papers on gradient noise injection, such as Jacobs et al. (1991). This omission makes it difficult to understand the context of their work and how it relates to existing research. The absence of a thorough literature review is a serious oversight that undermines the credibility of the paper (Confidence Level: High). Furthermore, the paper does not provide sufficient details about the model architecture, hyperparameter settings, and other implementation details. This lack of transparency makes it challenging for other researchers to reproduce the results, which is a crucial aspect of scientific research (Confidence Level: High). Lastly, the paper does not compare the proposed method with other regularization techniques, such as dropout, weight decay, or data augmentation. This omission limits the understanding of how the proposed method interacts with other common regularization strategies and whether it offers any unique advantages (Confidence Level: High).

💡 Suggestions

To address the identified weaknesses and enhance the paper's overall quality, several concrete and actionable improvements are recommended. First, the authors should conduct a more thorough exploration of the existing literature on gradient noise injection. This would involve citing and discussing relevant works, such as Jacobs et al. (1991), and clearly positioning their method within this context. The authors should explicitly state how their approach differs from existing techniques and provide a theoretical or empirical justification for the specific noise schedule they use. For instance, they could compare their exponential decay schedule with other schedules, such as linear decay or adaptive noise levels, to demonstrate its superiority or unique advantages. Second, the authors should perform a more comprehensive evaluation of their method. This should include a broader range of datasets, including larger and more diverse text corpora, to demonstrate the generalizability of their approach. Additionally, the authors should report error bars and conduct statistical significance tests to validate the observed improvements in training and validation losses. This would provide a more robust and reliable assessment of the method's effectiveness. Third, the authors should provide more detailed information about the model architecture, hyperparameter settings, and other implementation details. This should include the specific architecture of the small language models used, the size of the training and validation datasets, the learning rate, the batch size, and the optimization algorithm. The authors should also consider releasing the code and trained models to facilitate reproducibility and further research. Fourth, the authors should compare their method with other regularization techniques. This could involve experiments where the proposed gradient noise injection is combined with dropout, weight decay, or data augmentation. The authors should analyze how these combinations affect the training dynamics, convergence speed, and final performance of the models. This would provide a more nuanced understanding of the method's strengths and limitations. Finally, the authors should discuss the computational cost of their method and compare it to other regularization techniques. This would help readers understand the practical implications of using gradient noise injection, especially in resource-constrained environments.

❓ Questions

1. Could the authors provide more details about the model architecture, hyperparameter settings, and other implementation details? For example, what is the specific architecture of the small language models used in the experiments, and how were the hyperparameters chosen? 2. How does the proposed method compare to other regularization techniques, such as dropout, weight decay, and data augmentation? Are there any synergistic effects when combining gradient noise injection with these methods? 3. What is the computational cost of the proposed method compared to other regularization techniques? Is the method computationally efficient, and how does it scale with larger models and datasets? 4. Could the authors provide a more detailed analysis of the impact of different noise schedules on the training dynamics and final performance of the model? For instance, how do linear decay or adaptive noise levels compare to the exponential decay schedule used in the paper? 5. How do the authors justify the choice of the initial noise level and the decay rate? Are these parameters tuned, and if so, what is the tuning process?

📊 Scores

Soundness:1.5
Presentation:1.0
Contribution:1.0
Rating: 2.0

AI Review from ZGCA

ZGCA Review available after:
--d --h --m --s

📋 AI Review from ZGCA will be automatically processed

📋 Summary

The paper studies gradient noise injection (GNI) for training small language models (LMs). The method adds zero-mean Gaussian noise to gradients during backpropagation with an exponentially decaying schedule σ_t = σ_0 · γ^t (Section 4.2), aiming to regularize training, improve robustness, and aid generalization. Experiments on character-level datasets (shakespeare_char, enwik8, text8) compare a baseline (no gradient noise) to GNI using AdamW (Section 5). The authors report improved training dynamics, faster convergence, and lower training/validation loss with GNI.

✅ Strengths

  • Simplicity and ease of integration: adding annealed Gaussian noise to gradients is lightweight and requires minimal code changes (Section 4.3).
  • Clear description of the core mechanism and schedule (Section 4.1–4.2).
  • Evaluation on three character-level corpora with stated training iterations and optimizer details (Section 5).
  • Motivation aligns with theory that noise can promote flatter minima and regularization (Sections 1 and 4).

❌ Weaknesses

  • Limited novelty: annealed gradient noise is known (Neelakantan et al., 2015) and the exponential decay schedule is a standard choice; no new algorithmic insight, analysis, or theory is provided beyond applying it to small LMs.
  • Underpowered evaluation: comparison only to a non-regularized baseline (Section 6); lacks comparisons to standard regularizers (dropout, weight decay, label smoothing) or optimizers (SGD vs. AdamW), making it hard to assess practical utility.
  • Marginal and mixed gains: Table 1 shows very small differences and even worse validation on enwik8 (1.0048 baseline vs. 1.0067 GNI) and worse training loss on text8 (0.9978 vs. 1.0041), contradicting claims of consistent improvements in Sections 6.1–6.3.
  • Insufficient statistical rigor: no multiple seeds, standard deviations, or confidence intervals; only single-run loss curves and point estimates (Section 6).
  • Reproducibility gaps: missing full architectural specifications of the "small LMs" (layers, hidden sizes, context length, vocab), batch sizes, and random seeds (Section 5); ablation settings (σ0, γ) are listed but results are not reported.
  • Claims not fully substantiated: faster convergence and training stability are asserted (Sections 6.1, 6.2) without quantitative convergence metrics or stability measures; only qualitative discussion of curves.
  • Presentation issues: missing citations denoted by '?' in Section 3 (e.g., GPT-3, dropout), and no standard LM metrics (bits-per-character/perplexity) on these text datasets.
  • No analysis of interactions with common training components (e.g., gradient clipping, weight decay in AdamW, noise scaling with parameter/gradient norms) or computational overhead/wall-clock impact.

❓ Questions

  • Please provide full architectural details of the "small language models": model family (Transformer/RNN), number of layers, hidden size, attention heads, context length, vocab size/tokenization (character-level confirmation), positional encoding, and parameter counts per dataset.
  • What batch sizes and random seeds were used? How many runs per setting? Please report mean ± std over at least 3 seeds for all metrics.
  • Can you include competitive baselines: dropout, weight decay (explicitly controlled), label smoothing, and perhaps stochastic depth or data augmentation where appropriate? How does GNI compare and/or combine with these?
  • Why assess only loss? For language modeling on enwik8/text8, please report bits-per-character (BPC) and/or perplexity on validation/test. Do conclusions hold under these standard metrics?
  • The schedule is σ_t = σ_0 γ^t. Did you explore alternative schedules (e.g., inverse square root, cosine, linear) or adaptive/data-dependent schedules? Please include ablations for σ_0 and γ with quantitative results.
  • How is the noise applied in practice? Element-wise per-parameter Gaussian with fixed σ_t across all parameters, or scaled by parameter/gradient magnitude (e.g., proportional to ||g_t|| or parameter scale)? Is noise added before/after gradient clipping and before/after AdamW’s decoupled weight decay?
  • How does GNI interact with AdamW’s adaptive statistics? Would results differ with SGD/Momentum? Please compare optimizers.
  • Convergence: can you quantify “faster convergence” (e.g., iterations/wall-clock to reach a fixed validation loss threshold) and report wall-clock overhead introduced by noise generation?
  • On enwik8, Table 1 shows worse validation loss with GNI. Can you reconcile this with the claim of consistent gains, and analyze when GNI helps or hurts?
  • Did you control for hyperparameter tuning fairness? Were learning rates, weight decay, and other hyperparameters separately tuned for each method? Please describe the tuning protocol.

⚠️ Limitations

  • Scope limited to small, character-level LMs on three datasets; does not cover larger models, subword tokenization, or diverse domains, as acknowledged in Section 6.5.
  • Reported gains are small and sometimes negative (Table 1), suggesting sensitivity to schedule/hyperparameters and limited robustness across tasks.
  • No statistical analysis across seeds; results may not be reliable or generalizable.
  • No comparison to strong regularization baselines; unclear practical advantage versus standard techniques like dropout or weight decay.
  • Potential instability: poorly tuned noise may hinder convergence or degrade performance; interaction with adaptive optimizers and gradient clipping is unexamined.
  • No analysis of compute efficiency: overhead of noise generation and any wall-clock impact are not reported.
  • Potential negative impacts appear minimal for this technique, but improper use could inadvertently degrade models deployed in resource-constrained settings if users assume universal benefits.

🖼️ Image Evaluation

Cross-Modal Consistency: 30/50

Textual Logical Soundness: 18/30

Visual Aesthetics & Clarity: 16/20

Overall Score: 64/100

Detailed Evaluation (≤500 words):

Image-first synopsis (visual ground truth)

• Figure 1/(a): Training loss vs. iteration (shakespeare_char); legend lists Baseline and several noise schedules; all curves decrease and largely overlap.

• Figure 1/(b): Training loss vs. iteration (enwik8); similar axes/legend; curves nearly indistinguishable.

• Figure 1/(c): Training loss vs. iteration (text8); same pattern; overlapping decreases.

• Figure 1 synopsis: Three per‑dataset training‑loss trajectories; no clear separation between methods.

• Figure 2/(a): Validation loss vs. iteration (shakespeare_char); dip then slight rise; multiple schedules overlap.

• Figure 2/(b): Validation loss vs. iteration (enwik8); smooth decrease; overlaps.

• Figure 2/(c): Validation loss vs. iteration (text8); smooth decrease; overlaps.

• Figure 2 synopsis: Validation‑loss curves for three datasets; method lines largely coincide with baseline.

• Table 1: Final train/val losses; GNI better on val for text8 (very small), worse on val for enwik8; train worse for enwik8 and text8.

1. Cross-Modal Consistency

• Major 1: Training‑loss improvement claim conflicts with Table 1 (enwik8/text8 train loss higher for GNI). Evidence: Sec. 6.1 “consistently achieve lower training loss”; Table 1 enwik8 0.9323 vs 0.9379; text8 0.9978 vs 1.0041.

• Major 2: Validation‑loss claim conflicts with Table 1 for enwik8 (worse with GNI). Evidence: Sec. 6.2 “consistently achieve lower validation loss”; Table 1 enwik8 1.0048 vs 1.0067.

• Major 3: “Faster convergence” repeatedly claimed but no quantitative support (no time/iteration‑to‑target or speed metric). Evidence: Abstract “improves … convergence speed”; no figure/table reports speed.

• Minor 1: Fig. 2 subfigure labels appear out of order (b shown before a), risking dataset confusion. Evidence: Fig. 2 displays “(b) enwik8” preceding “(a) shakespeare_char”.

• Minor 2: Overlapping curves make claimed separation visually unverifiable across figs.

2. Text Logic

• Major 1: Broken/placeholder citations reduce credibility of background. Evidence: Sec. 3 “GPT‑3 (?)”, “dropout (?)”, “SGD with noise (?)”.

• Minor 1: Method lacks concrete model specs (sizes/params), hindering reproducibility and interpretation of “small” models.

• Minor 2: Notation inconsistency: bold σt in Eq. (4.2) for a scalar.

3. Figure Quality

• Minor 1: Legends/lines thin and tightly overlapping; difficult to discern differences at print size.

• Minor 2: No error bands or multiple‑run variability; undermines robustness claims visually.

• Minor 3: Fig. 2 label ordering/layout slightly confusing; ensure a→b→c left‑to‑right.

Key strengths:

• Clear, simple method and easily implementable schedule.

• Consistent experimental protocol across three datasets.

Key weaknesses:

• Central claims (lower losses, faster convergence) not supported by Table 1 and indistinct plots.

• Broken citations and sparse model details.

• Figures lack statistical depiction (variance/error bars) and have minor labeling issues.

📊 Scores

Originality:1
Quality:2
Clarity:2
Significance:1
Soundness:2
Presentation:2
Contribution:2
Rating: 3

AI Review from SafeReviewer

SafeReviewer Review available after:
--d --h --m --s

📋 AI Review from SafeReviewer will be automatically processed

📋 Summary

This paper introduces a method for enhancing the training of small language models (SLMs) by injecting Gaussian noise into the gradients during the backward pass. The core idea is to improve the robustness and generalization of these models, which often struggle with limited capacity and overfitting. The authors propose a noise schedule that decreases the noise level over time, starting with a higher initial noise level to encourage exploration in the parameter space and gradually reducing it to allow for convergence. The method is implemented by adding noise to the gradients before the optimizer updates the model parameters, requiring minimal changes to existing training pipelines. The experimental evaluation focuses on three character-level datasets—shakespeare_char, enwik8, and text8—using a 2-layer MLP with 2 heads and 512 hidden dimensions. The primary metrics used are training and validation loss. The results show that the proposed gradient noise injection technique leads to lower training and validation losses compared to a baseline model trained without noise injection. The authors argue that this method helps the model escape sharp local minima and converge to flatter regions of the loss landscape, which are associated with better generalization. While the paper demonstrates improvements in loss metrics, it lacks a comprehensive evaluation of the method's impact on downstream task performance and does not provide a detailed analysis of the model's performance on the test set. Furthermore, the paper does not include a comparison with other regularization techniques, such as dropout or label smoothing, which makes it difficult to assess the relative effectiveness of the proposed approach. The paper also suffers from some issues with writing quality, including grammatical errors and unclear phrasing, which detract from the overall clarity of the work. Despite these limitations, the paper presents a simple and lightweight method that shows promise for improving the training of small language models.

✅ Strengths

I find the core idea of injecting noise into the gradients to be a compelling approach to improving the training of small language models. The method is elegantly simple, requiring minimal modifications to existing training pipelines, and the use of a decaying noise schedule is a logical way to balance exploration and convergence. The authors' focus on small language models is also timely, given the increasing interest in more efficient and deployable models. The experimental results, while limited in scope, do demonstrate a clear improvement in training and validation loss when using the proposed gradient noise injection technique. This suggests that the method is indeed effective at regularizing the training process and preventing overfitting, at least as measured by these loss metrics. The paper also provides a clear description of the method, making it easy to understand and implement. The use of a 2-layer MLP with a relatively small number of parameters (approximately 2 million) is appropriate for the focus on small language models. The paper's motivation is also well-articulated, highlighting the challenges of training small models and the need for effective regularization techniques. The authors' claim that the method helps the model escape sharp local minima and converge to flatter regions of the loss landscape is also a plausible explanation for the observed improvements, although further analysis would be needed to confirm this. Overall, the paper presents a promising technique that warrants further investigation.

❌ Weaknesses

After a thorough examination of the paper, I've identified several significant weaknesses that impact the validity and generalizability of the findings. First, the paper lacks a clear and detailed description of the model architecture used in the experiments. While the authors mention a 2-layer MLP with 2 heads and 512 hidden dimensions, they do not specify the number of layers, the size of the embedding dimensions, or the number of attention heads. This lack of detail makes it difficult to assess the complexity of the model and compare it to other models in the literature. Furthermore, the paper does not provide a clear rationale for the specific model architecture chosen, leaving me to wonder if the observed improvements are specific to this particular configuration. This is a significant issue, as the effectiveness of regularization techniques can be highly dependent on the model architecture. The paper also fails to provide a comprehensive evaluation of the proposed method. The primary metrics used are training and validation loss, which, while useful for assessing convergence, do not directly measure the model's ability to generalize to unseen data or perform downstream tasks. The paper does not include any evaluation on a held-out test set, which is a crucial step in assessing the model's generalization capabilities. Moreover, the paper does not compare the proposed method with other common regularization techniques, such as dropout or label smoothing. This lack of comparison makes it difficult to determine whether the proposed method is superior to existing approaches or if it simply provides a marginal improvement. The absence of these comparisons significantly limits the paper's contribution. The paper also suffers from a lack of novelty. The idea of adding noise to gradients during training has been explored in prior work, and the paper does not adequately differentiate its approach from these existing methods. The authors do not provide a detailed analysis of the specific noise schedule used, nor do they compare it to other noise schedules. The paper also does not address the potential negative impacts of gradient noise injection, such as instability or slower convergence, which have been observed in other studies. This lack of discussion is a significant oversight. The experimental evaluation is also limited in scope. The paper only evaluates the method on three character-level datasets, which may not be representative of the broader range of language modeling tasks. The datasets used are also relatively small, which may limit the generalizability of the findings to larger datasets. The paper also lacks a detailed analysis of the hyperparameter tuning process. The authors mention experimenting with different initial noise levels and decay rates, but they do not provide a systematic analysis of how these parameters affect the model's performance. This lack of analysis makes it difficult to determine the optimal hyperparameter settings for the proposed method. Finally, the paper suffers from several issues with writing quality, including grammatical errors, typos, and unclear phrasing. These issues detract from the overall clarity of the paper and make it difficult to understand the authors' arguments. For example, the paper uses the phrase "Gaussan noise" instead of "Gaussian noise", and the word "flater" instead of "flatter". These errors, while seemingly minor, undermine the credibility of the work. In summary, the paper's weaknesses, including the lack of model details, limited evaluation, lack of novelty, limited experimental scope, and writing quality issues, significantly impact the validity and generalizability of the findings. These issues need to be addressed before the paper can be considered a significant contribution to the field.

💡 Suggestions

To address the identified weaknesses, I recommend several concrete improvements. First, the authors should provide a detailed description of the model architecture used in the experiments, including the number of layers, the size of the embedding dimensions, the number of attention heads, and any other relevant architectural details. This would allow for a more accurate assessment of the model's complexity and facilitate comparisons with other models. The authors should also provide a clear rationale for the chosen architecture, explaining why it is appropriate for the task at hand. Second, the authors should conduct a more comprehensive evaluation of the proposed method. This should include evaluating the model's performance on a held-out test set, as well as on downstream tasks that are relevant to the specific datasets used. The authors should also compare the proposed method with other common regularization techniques, such as dropout and label smoothing, to determine its relative effectiveness. This comparison should include an analysis of the computational overhead of each method. Third, the authors should provide a more detailed analysis of the proposed noise injection method. This should include a discussion of the specific noise schedule used, as well as a comparison with other noise schedules. The authors should also address the potential negative impacts of gradient noise injection, such as instability or slower convergence, and discuss how their method mitigates these issues. Fourth, the authors should expand the scope of their experimental evaluation. This should include evaluating the method on a wider range of datasets, including larger datasets and datasets with different characteristics. The authors should also consider evaluating the method on different model architectures, to determine its generalizability. Fifth, the authors should conduct a more detailed analysis of the hyperparameter tuning process. This should include a systematic analysis of how the initial noise level and decay rate affect the model's performance, as well as a discussion of how to choose the optimal hyperparameter settings. The authors should also consider using a more sophisticated hyperparameter optimization technique. Finally, the authors should carefully proofread the paper to correct any grammatical errors, typos, and unclear phrasing. This would significantly improve the clarity and credibility of the work. The authors should also consider adding visualizations of the training dynamics, such as loss curves and gradient magnitudes, to provide more insight into the behavior of the proposed method. These visualizations would help to illustrate the impact of the noise injection on the optimization process. By addressing these weaknesses, the authors can significantly improve the quality and impact of their work.

❓ Questions

After reviewing the paper, I have several questions that I believe are crucial for a deeper understanding of the proposed method and its implications. First, I am curious about the specific mechanism by which the gradient noise injection leads to lower training and validation losses. While the authors suggest that it helps the model escape sharp local minima and converge to flatter regions of the loss landscape, I would like to see more concrete evidence for this claim. Are there any visualizations of the loss landscape that support this interpretation? Second, I am interested in the sensitivity of the proposed method to the choice of hyperparameters, particularly the initial noise level and the decay rate. The authors mention experimenting with different values for these parameters, but they do not provide a detailed analysis of how these parameters affect the model's performance. What is the optimal range for these parameters, and how can they be tuned for different datasets and model architectures? Third, I am curious about the computational overhead of the proposed method. While the authors claim that it is lightweight and requires minimal modifications to existing training pipelines, I would like to see a more detailed analysis of the computational cost. How does the computational cost of the proposed method compare to other regularization techniques, such as dropout or label smoothing? Fourth, I am interested in the generalizability of the proposed method to larger models and datasets. The experiments in the paper are limited to small language models and relatively small datasets. How does the proposed method perform on larger models and datasets, and are there any scalability issues that need to be addressed? Fifth, I am curious about the potential negative impacts of gradient noise injection. While the authors focus on the positive effects of the method, I would like to know more about the potential drawbacks. Can the noise injection lead to instability or slower convergence in certain situations, and how can these issues be mitigated? Finally, I would like to understand the authors' choice of a linear decay schedule for the noise level. Is there a theoretical justification for this choice, or is it simply a practical heuristic? Have the authors experimented with other noise schedules, and if so, how did they compare to the linear decay schedule? These questions are crucial for a more complete understanding of the proposed method and its potential applications.

📊 Scores

Soundness:1.75
Presentation:1.75
Contribution:1.75
Confidence:4.25
Rating: 2.0

Keywords

Click the button to extract keywords

Insights

Click the button to extract insights
Version Management
Version 1 ⚠️ Not latest
📋 Version History
Citation Tools

📝 Cite This Paper