2510.0052 Conformal Prediction as Bayesian Quadrature for Risk Control v1

🎯 ICAIS2025 Accepted Paper
Back to Paper List

Paper Content

📄 Open in New Tab

🎓 Meta Review & Human Decision

Decision:

Accept

Meta Review:

AI Review from DeepReviewer

AI Review available after:
--d --h --m --s

📋 AI Review from DeepReviewer will be automatically processed

📋 Summary

This paper introduces a novel framework for conformal prediction, aiming to provide rigorous, data-conditioned, and distribution-free risk guarantees. The core idea revolves around constructing an upper bound on the expected loss by integrating over the quantile function of the loss distribution. The authors propose an aggregated loss function, denoted as L+, which is defined as a weighted sum of individual losses and a worst-case loss, with the weights drawn from a Dirichlet distribution. This construction ensures that the probability of L+ exceeding a certain threshold is bounded, providing a basis for risk control. The framework is presented as a generalization of existing conformal prediction methods, specifically Split Conformal Prediction (SCP) and Conformal Risk Control (CRC), which are shown to be special cases within this broader approach. A key contribution is the introduction of a High Posterior Density (HPD) decision rule, which leverages the full posterior distribution of L+, approximated through Monte Carlo sampling, to make more informed decisions. The authors claim that this HPD rule offers improved risk control and utility compared to methods relying solely on the posterior mean or uniform concentration bounds. The empirical validation of the proposed method is conducted on synthetic datasets, including binomial loss and heteroskedastic regression tasks. The results demonstrate the effectiveness of the HPD rule in achieving risk control with zero empirical failure rate in these synthetic settings. The paper argues that the proposed framework is particularly relevant for high-stakes applications where reliable uncertainty quantification is crucial. However, the paper's presentation and motivation have several shortcomings that need to be addressed to fully realize its potential. The lack of clear definitions for key terms, the absence of real-world validation, and the somewhat unclear connection to standard Bayesian quadrature techniques are significant limitations. Despite these issues, the paper presents an interesting approach to conformal prediction that warrants further investigation and refinement.

✅ Strengths

Despite the identified weaknesses, the paper does present some notable strengths. The core idea of constructing an aggregated loss function, L+, using a Dirichlet distribution to provide probabilistic risk guarantees is a novel approach within the conformal prediction framework. The paper successfully demonstrates that well-known conformal methods, such as Split Conformal Prediction (SCP) and Conformal Risk Control (CRC), can be viewed as special cases within this broader framework. This unification of existing methods provides a valuable perspective and highlights the generality of the proposed approach. The introduction of the High Posterior Density (HPD) decision rule is another strength. By leveraging the full posterior distribution of the aggregated loss, the HPD rule offers a potentially more efficient way to control risk compared to methods that rely on simpler summary statistics. The empirical results, although limited to synthetic datasets, do show that the HPD rule achieves risk control with zero empirical failure rate, suggesting its potential for practical applications. The paper's focus on providing rigorous uncertainty guarantees in high-stakes settings is also a significant strength. The authors correctly identify the need for reliable risk control in applications where misestimation can have serious consequences. The paper's attempt to address this need by providing data-conditioned risk guarantees is a valuable contribution to the field of conformal prediction. Finally, the paper is generally well-organized and clearly written, making it accessible to readers with a background in conformal prediction and risk control. The mathematical formulations are presented in a clear and concise manner, and the experimental setup is described in sufficient detail to allow for reproducibility. These strengths suggest that the paper has the potential to make a meaningful contribution to the field, provided that the identified weaknesses are addressed.

❌ Weaknesses

The paper suffers from several significant weaknesses that undermine its clarity, motivation, and practical relevance. Firstly, the presentation of the paper is poor, characterized by vague statements and a lack of clear definitions. The introduction is particularly problematic, with terms like "rigorous uncertainty guarantees," "interpretable and data-conditional risk guarantees," "distribution-free risk guarantees," and "high-stakes black-box settings" used without explicit definitions or connections to specific mathematical formulations. This lack of precision makes it difficult to understand the paper's core contribution and its relation to existing work. For instance, the term "interpretable and data-conditional risk guarantees" is never formally defined, leaving the reader to guess at its meaning. This lack of clarity extends to the motivation for the proposed method. While the introduction mentions the limitations of traditional conformal prediction methods, it fails to explicitly connect these limitations to the specific design choices of the Bayesian quadrature approach. The motivation for using a Dirichlet distribution for weights and the specific form of the aggregated loss function could be more clearly articulated. The paper also makes claims that are not fully justified or are potentially contradictory. For example, the authors state that traditional conformal prediction methods are based on frequentist principles and often lead to overly conservative or optimistic decisions. However, the paper does not clearly explain why this is a problem, especially given that valid prediction sets are the goal. The paper also states that methods relying solely on the posterior mean might underestimate uncertainty, and that approaches based on simple order statistics can yield high failure rates. While these statements may be true, the paper does not fully explain why these are undesirable, especially in the context of high-stakes applications where reliable risk control is paramount. The paper's claim of leveraging "Bayesian quadrature" is also problematic. While the method uses Monte Carlo sampling to approximate a posterior distribution, the connection to standard Bayesian quadrature techniques is not made explicit. The paper does not provide a clear definition of Bayesian quadrature, nor does it explain how the proposed method relates to existing Bayesian quadrature techniques. This lack of clarity makes it difficult to assess the novelty and significance of the proposed approach. Furthermore, the paper's motivation is linked to "classical results in tolerance regions and order statistics," but these results are never explicitly cited or described. This lack of detail makes it difficult to understand the theoretical underpinnings of the proposed method. The paper's core idea of controlling risk by ensuring that Pr(L+ ≤ α) ≥ β is not clearly explained. While the intuition is present, the paper does not provide a clear explanation of why this condition is important or how it relates to the overall goal of risk control. The paper also claims to introduce a novel high posterior density (HPD) decision rule, but the use of the term "posterior" is misleading, as the method relies on Monte Carlo sampling to approximate a distribution rather than a true Bayesian posterior. The paper's experimental validation is also a major weakness. The experiments are limited to synthetic datasets, specifically binomial loss and heteroskedastic regression tasks. While these experiments demonstrate the core mechanics of the method, they do not address the practical challenges of applying the method to complex, real-world data with potential noise and outliers. The absence of real-world validation makes it difficult to assess the practical relevance of the proposed method. Finally, the paper does not provide sufficient guidance on the selection of the confidence level β. The paper does not discuss how the choice of β affects the trade-off between risk control and utility, nor does it provide any recommendations on how to select an appropriate value of β in practice. This lack of guidance makes it difficult for practitioners to apply the method in real-world settings. These weaknesses, taken together, significantly undermine the paper's clarity, motivation, and practical relevance. The lack of clear definitions, the absence of real-world validation, and the somewhat unclear connection to standard Bayesian quadrature techniques are significant limitations that need to be addressed.

💡 Suggestions

To address the identified weaknesses, I recommend several concrete improvements. First and foremost, the paper needs a major rewrite to improve its clarity and precision. The introduction should be revised to provide clear definitions for all key terms, including "rigorous uncertainty guarantees," "interpretable and data-conditional risk guarantees," "distribution-free risk guarantees," and "high-stakes black-box settings." These definitions should be connected to specific mathematical formulations within the paper. The motivation for the proposed method should be strengthened by explicitly linking the shortcomings of existing methods to the specific design choices of the Bayesian quadrature approach. The paper should clearly explain why the limitations of traditional conformal prediction methods, such as overly conservative or optimistic decisions, are problematic in the context of high-stakes applications. The paper should also clarify the connection between the proposed method and standard Bayesian quadrature techniques. This could involve providing a formal definition of Bayesian quadrature and explaining how the proposed method relates to existing Bayesian quadrature techniques. If the method deviates from standard definitions, this should be made clear, and the authors should provide a more precise description of their approach. The paper should also provide more detail on the "classical results in tolerance regions and order statistics" that motivate the approach. This could involve citing relevant works and explaining how these results relate to the construction of the aggregated loss function. The paper should also provide a more detailed explanation of the intuition behind the condition Pr(L+ ≤ α) ≥ β. This explanation should be connected to the overall goal of risk control and should be made accessible to readers with a background in conformal prediction. The paper should also clarify the use of the term "posterior" in the context of the HPD rule. The authors should acknowledge that the method relies on Monte Carlo sampling to approximate a distribution rather than a true Bayesian posterior and should avoid using misleading terminology. The experimental validation of the method should be significantly improved. The paper should include experiments on real-world datasets, particularly in high-stakes applications where risk control is crucial, such as medical diagnosis or financial forecasting. These experiments should include a comparison of the proposed method with existing methods, such as SCP and CRC, in terms of both risk control and utility. The authors should also discuss the challenges of applying the method to real-world data, such as the presence of noise, outliers, and violations of the underlying assumptions of the model. The paper should also provide more guidance on the selection of the confidence level β. This could involve a sensitivity analysis of the method to different values of β, with a focus on the trade-off between risk control and utility. The authors should discuss how the choice of β affects the width of the prediction intervals and the implications of these changes for decision-making in high-stakes applications. The authors should also provide recommendations on how to select an appropriate value of β in practice, based on the specific requirements of the application and the desired balance between risk control and utility. Finally, the paper should be carefully proofread to ensure that there are no grammatical errors or typos. These suggestions, if implemented, would significantly improve the clarity, motivation, and practical relevance of the paper.

❓ Questions

Several key questions arise from my analysis of this paper. Firstly, how does the computational cost of the proposed HPD rule compare to that of existing methods like SCP and CRC, especially for large datasets? The paper does not provide a detailed analysis of the computational complexity of the HPD rule, and it would be important to understand the practical implications of using this method in real-world settings. Secondly, what are the potential applications of the proposed method in real-world scenarios, and how does it perform in these settings compared to existing methods? The paper's experiments are limited to synthetic datasets, and it would be crucial to evaluate the method's performance on real-world data with complex characteristics. Thirdly, how sensitive is the performance of the HPD rule to the choice of the confidence level β, and what are the guidelines for selecting an appropriate value of β in practice? The paper does not provide sufficient guidance on the selection of β, and it would be important to understand how this parameter affects the trade-off between risk control and utility. Fourthly, what is the precise connection between the proposed method and standard Bayesian quadrature techniques? The paper claims to leverage Bayesian quadrature, but the connection is not made explicit, and it would be important to clarify this aspect of the method. Fifthly, what are the specific "classical results in tolerance regions and order statistics" that motivate the approach, and how do these results relate to the construction of the aggregated loss function? The paper mentions these results but does not provide any details, and it would be important to understand the theoretical underpinnings of the method. Finally, what is the intuition behind the condition Pr(L+ ≤ α) ≥ β, and how does this condition relate to the overall goal of risk control? The paper states this condition but does not provide a clear explanation of its importance. These questions highlight key uncertainties and areas where further clarification is needed to fully understand the proposed method and its potential impact.

📊 Scores

Soundness:2.5
Presentation:2.25
Contribution:2.5
Rating: 4.5

AI Review from ZGCA

ZGCA Review available after:
--d --h --m --s

📋 AI Review from ZGCA will be automatically processed

📋 Summary

The paper proposes a Bayesian quadrature formulation of conformal risk control. Given calibration losses ℓ_1,...,ℓ_n, the authors define an aggregated loss L^{+} = ∑_{i=1}^{n+1} U_i ℓ_{(i)} with U ~ Dir(1,...,1) over the order statistics augmented by ℓ_{(n+1)} = B, and select a threshold λ such that Pr(L^{+} ≤ α) ≥ β (Sections 1–4). This framework is claimed to recover Split Conformal Prediction (SCP) via an order-statistic criterion and Conformal Risk Control (CRC) via the posterior mean E[L^{+}] = (∑ℓ_i + B)/(n+1) (Sections 3–4). The paper introduces a High Posterior Density (HPD) rule that uses Monte Carlo sampling of U to enforce the inequality and compares HPD with SCP, CRC, and a uniform concentration baseline (RCPS) on synthetic binomial-loss and heteroskedastic regression tasks (Sections 5–6). Empirically, HPD attains 0% failure with better utility than RCPS and less optimism than CRC.

✅ Strengths

  • Conceptually novel unification: casting conformal risk control in a Bayesian quadrature/Bayesian bootstrap form with Dirichlet(1,...,1) weights over ordered calibration losses and an appended bound B (Sections 1–2, 4).
  • Clear connection to known methods: CRC is recovered via the posterior mean (Equation for λ_CRC in Sections 3–4); SCP is argued to arise from a fixed order-statistic criterion (Equations for λ_SCP in Sections 4–5).
  • Practical HPD rule: a simple Monte Carlo decision rule using the full posterior of L^{+} to trade off risk and utility through β (Section 4).
  • Empirical promise: on synthetic binomial and regression setups, HPD achieves 0% failure with improved utility relative to RCPS and less optimism than CRC (Section 6).
  • Sensitivity analyses indicate the method adapts with β and calibration size n (Section 6).

❌ Weaknesses

  • Guarantee gap: The main guarantee is framed as Pr(L^{+} ≤ α) ≥ β with probability taken over Dirichlet weights conditional on the observed calibration losses (Sections 1–2, 4). The paper does not provide a theorem that translates this posterior statement into the usual frequentist, distribution-free risk control for out-of-sample data. Without such a result, the claims of distribution-free risk control remain unsubstantiated.
  • SCP/CRC as special cases: While CRC via E[L^{+}] is straightforward, the paper does not provide a precise derivation establishing SCP as an instance of the proposed framework (e.g., specifying β and using Dirichlet spacing properties to recover the classic order-statistic rule) (Sections 3–4).
  • Experimental scope: Evaluation is exclusively synthetic (binomial loss and heteroskedastic regression) (Sections 5–6). Despite claims about high-stakes, black-box settings, there is no real-data validation or analysis of robustness to model misspecification beyond the bounded-loss B augmentation.
  • Methodological clarity: Important implementation details are insufficiently specified. Examples include the exact definition of the per-sample loss for each task, the mapping between λ and ℓ_i across tasks, how B is chosen beyond B=1 in the binomial experiment, and computational cost of HPD (N_dirichlet, convergence diagnostics) (Sections 4–6).
  • Potential SCP mismatch: The SCP decision rule is presented as λ ≥ ℓ_{(⌈(n+1)(1−α)⌉)} (Section 4). It is not fully clear that this is the standard SCP application for the stated losses and tasks; the poor SCP performance reported in Section 6 suggests the need to carefully validate that the SCP baseline is implemented in its conventional manner for the given loss definitions.

❓ Questions

  • Formal guarantee: Can you provide a theorem and proof that relate the posterior probability condition Pr(L^{+} ≤ α) ≥ β (over U|data) to a frequentist, distribution-free guarantee on the induced decision rule’s risk on future data? Under what assumptions, and how does β translate to a bound on failure probability?
  • SCP as a special case: Precisely state and prove the conditions (on β, n, and properties of Dirichlet spacings) under which the HPD framework reduces to the classic SCP order-statistic rule you present in Section 4.
  • Choice of B: Beyond setting ℓ_{(n+1)} = B = 1 in the binomial experiment, how should B be set in general? If the loss is unbounded or heavy-tailed, what practical guidance or robust alternatives do you recommend? How sensitive is HPD to misspecification of B?
  • Loss and λ mapping: For each task, please precisely define ℓ(z, λ), explain how monotonicity in λ is verified, and detail how calibration losses are computed from data and λ.
  • HPD computation: What is the computational cost (time, memory) as a function of n and N_dirichlet? Do you have diagnostics for Monte Carlo error (e.g., variance estimates, stopping criteria) and guidance for choosing N_dirichlet?
  • Baselines: Please clarify the exact SCP and CRC implementations for each task, including any discretization of λ and how candidate grids influence guarantees. Could the SCP performance be an artifact of grid coarseness or an unconventional mapping from ℓ to λ?
  • Generality: Can you include at least one real-data experiment (e.g., a standard regression/classification benchmark) to show robustness beyond synthetic settings, and report utility-risk trade-offs along with calibration-size sensitivity?
  • Unification: Beyond CRC and SCP, can your framework connect to or subsume other risk-controlling or posterior-based methods (e.g., Bayesian bootstrap, RCPS variants)?

⚠️ Limitations

  • Reliance on i.i.d. calibration losses and bounded-loss assumption ℓ ∈ [0, B]; extensions to non-i.i.d. settings and heavy tails are identified but not addressed (Sections 2, 6–7).
  • Posterior vs. frequentist control: The core guarantee is Bayesian/posterior in nature; without a frequentist translation, users may overinterpret it as a conformal-style finite-sample guarantee (Sections 1–4).
  • Computational overhead from Monte Carlo Dirichlet sampling may be nontrivial for large n or many candidate λ; no complexity analysis or acceleration is provided (Sections 4–5).
  • Empirical validation limited to synthetic data; robustness and practical utility in real high-stakes domains remain untested (Sections 5–6).
  • Discretization of λ: Decision rules depend on grid resolution; trade-offs and potential biases from discretization are not fully analyzed (Sections 5–6).

🖼️ Image Evaluation

Cross‑Modal Consistency: 31/50

Textual Logical Soundness: 16/30

Visual Aesthetics & Clarity: 10/20

Overall Score: 57/100

Detailed Evaluation (≤500 words):

1. Cross‑Modal Consistency

• Major 1: Table numbering/content inconsistent: Abstract says results are in Table 1, but Sec. 4 uses Table 1 for decision rules. Evidence: “These findings, summarized in Table 1” (Abstract) vs. “Table 1: Summary of decision rules” (Sec 4).

• Major 2: Results are referenced as “Table below” without numbering, breaking traceability. Evidence: “These results can be summarized in Table below” (Sec 6).

• Minor 1: Methods mention future figures (“histograms… boxplots”) not present anywhere, causing expectation–content mismatch. Evidence: “presented using detailed tables and figures (e.g., histograms… boxplots)” (Sec 5).

2. Text Logic

• Major 1: Claimed theoretical guarantees lack formal statements/proofs (no theorems/lemmas or parameterized bounds). Evidence: “We derive theoretical guarantees… ensure that Pr(L+ ≤ α) ≥ β” (Sec 1).

• Major 2: Unspecified “stochastic dominance by a Beta” claim without parameters or justification. Evidence: “aggregated loss stochastically dominates a Beta distribution” (Sec 2).

• Minor 1: “Recovers SCP and CRC as special cases” is asserted but not rigorously derived beyond informal formulas. Evidence: “recovers standard methods (SCP and CRC) as special cases” (Sec 1).

3. Figure Quality

• Major 1: Critical decision-rule table has malformed/ambiguous formulas (missing parentheses/bars), risking misinterpretation. Evidence: “CRC Σi=1nℓi+B/n+1≤α” and “RCPS ℓ+ √log(1/δ)/2n≤α” (Sec 4, Table 1).

• Major 2: Promised qualitative/quantitative figures are absent; none are labeled or numbered. Evidence: “presented using… figures” (Sec 5) but none included.

• Minor 1: Results tables lack captions/IDs; cannot be referenced unambiguously. Evidence: Two uncaptioned tables in Sec 6.

Key strengths:

  • Clear problem framing: risk control via a Bayesian quadrature view of conformal prediction.
  • Practical HPD rule with Monte Carlo Dirichlet sampling; experimental numbers are plausible and consistent within Sec 6.
  • Broad experimental coverage (binomial, heteroskedastic regression) and sensitivity analyses discussed.

Key weaknesses:

  • Missing formal theory (no precise statements/proofs for Pr(L+ ≤ α) ≥ β or the Beta dominance claim).
  • Severe table/numbering inconsistencies; ambiguous formulas in the core decision rule table.
  • Promised figures absent; results tables unnumbered and without captions.
  • Minor notation/typo issues in SCP expressions and spaced math (readability).

Recommendations:

  • Add a Theory section with formal lemmas (Dirichlet spacing lemma), theorem for risk guarantee with all parameters, and proofs.
  • Fix table numbering; split into Table 1 (decision rules) and Table 2–3 (results), each with clear captions.
  • Correct table formulas with explicit parentheses and definitions (e.g., (Σℓi + B)/(n+1) ≤ α; ȳ + sqrt(log(1/δ)/(2n)) ≤ α).
  • Include the referenced histograms/boxplots with readable axes/legends; ensure all visuals pass a “figure‑alone” test.

📊 Scores

Originality:3
Quality:2
Clarity:2
Significance:2
Soundness:2
Presentation:2
Contribution:2
Rating: 4

AI Review from SafeReviewer

SafeReviewer Review available after:
--d --h --m --s

📋 AI Review from SafeReviewer will be automatically processed

📋 Summary

The paper introduces a new framework for conformal risk control using Bayesian quadrature. The authors propose a new high posterior density (HPD) rule for selecting decision thresholds, which offers a more flexible and interpretable approach to conformal prediction. The method is validated on synthetic binomial loss and heteroskedastic regression tasks, demonstrating improved performance over existing methods.

✅ Strengths

I find several aspects of this paper to be commendable. The core strength lies in the novel perspective it offers on conformal risk control. By framing the problem through Bayesian quadrature, the authors provide a fresh interpretation of existing methods and introduce a new decision rule, the HPD rule, that leverages the full posterior distribution of the aggregated loss. This approach is theoretically grounded and provides a more flexible framework compared to methods that rely solely on order statistics or the posterior mean. The paper's ability to recover existing methods like SCP and CRC as special cases is also a significant strength. This demonstrates that the proposed framework is not just an alternative but a generalization of existing approaches, providing a unified perspective on conformal risk control. The empirical results, while limited in scope, provide some evidence that the proposed HPD rule can achieve a better trade-off between risk control and the size of the prediction sets compared to SCP and CRC. This is a practically relevant finding, as it suggests that the proposed method could lead to more efficient prediction sets in real-world applications. Furthermore, the paper is generally well-written and easy to follow, making the core ideas accessible to a broad audience. The authors clearly articulate their approach and provide sufficient details for the reader to understand the main contributions. The use of synthetic data allows for a controlled evaluation of the proposed method, highlighting its potential advantages in specific scenarios. The paper's focus on risk control, rather than just coverage, is also a strength, as it addresses a critical need in many practical applications where the consequences of misestimation can be severe. Finally, the paper's exploration of a Bayesian approach to conformal prediction is a valuable contribution to the field, which has traditionally been dominated by frequentist methods. This opens up new avenues for research and could lead to the development of more robust and adaptable uncertainty quantification methods.

❌ Weaknesses

Despite its strengths, this paper has several weaknesses that I have identified through my analysis. A primary concern is the lack of a rigorous theoretical justification for the Bayesian quadrature (BQ) approach in the context of conformal prediction. While the authors present BQ as a core motivation, the actual derivation of the method relies on the exchangeability assumption and properties of order statistics, similar to traditional conformal prediction. The paper does not explicitly detail the integration process over the quantile function using BQ, nor does it clearly explain how the BQ weights relate to the conformal prediction procedure. This makes the BQ perspective seem somewhat superficial, as the resulting method closely resembles standard conformal risk control techniques. The paper defines the aggregated loss $L^+$ as $L^+ = \sum_{i=1}^{n+1} U_i l(i)$, where $U \sim Dir(1,\dots,1)$, and states that it is motivated by classical results that guarantee $L^+$ stochastically dominates a Beta distribution. However, the connection between this construction and the BQ approach is not clearly established. The paper does not provide a detailed explanation of how the integral approximation in BQ leads to this specific form of $L^+$. This lack of clarity undermines the claim that the method is a principled application of BQ. My analysis also reveals that the paper's contribution beyond existing conformal risk control methods is not entirely clear. The proposed HPD rule, while novel, does not demonstrate a significant advantage over existing methods, particularly Randomized Concentration Prediction Sets (RCPS). The empirical results show that the HPD rule achieves similar risk control to RCPS, while being less conservative than Split Conformal Prediction (SCP) and Conformal Risk Control (CRC). However, the paper does not provide a compelling argument for why the HPD rule is superior to RCPS, especially considering the computational overhead of sampling from the posterior distribution. The paper also does not adequately address the limitations of the proposed method, particularly its reliance on the exchangeability assumption and the difficulty in handling complex decision sets. The authors acknowledge these limitations, but they do not provide a detailed discussion of how these limitations might impact the practical applicability of the method. The paper's experimental evaluation is also limited in scope. The experiments are conducted on synthetic datasets, which may not fully reflect the challenges of real-world applications. The paper does not include experiments on more complex datasets or with different types of models. This limits the generalizability of the findings and makes it difficult to assess the practical relevance of the proposed method. Furthermore, the paper lacks a detailed analysis of the computational cost of the proposed method. The HPD rule involves Monte Carlo sampling, which can be computationally expensive, especially for large datasets. The paper does not provide any information about the computational time required for the HPD rule compared to other methods, making it difficult to assess its practical feasibility. The paper also does not provide a clear explanation of how the method can be used to construct prediction sets in regression problems. The method focuses on selecting a threshold based on risk control, but it does not explicitly detail how this translates to the construction of prediction intervals in regression. This lack of clarity makes it difficult to understand the practical implications of the proposed method for regression tasks. Finally, I found that the paper's writing could be improved in several places. The notation is not always clearly defined, and some of the explanations are vague or difficult to follow. For example, the paper uses the notation $R(\lambda)$ without explicitly defining the dependence of the loss function on $\lambda$ in the definition of $R(\lambda)$. The paper also uses the notation $L^+$ before fully defining all its components. These issues make it more difficult for the reader to understand the core ideas of the paper.

💡 Suggestions

Based on the weaknesses I've identified, I have several suggestions for improving this paper. First, the authors should provide a more rigorous theoretical justification for the use of Bayesian quadrature in the context of conformal prediction. This should include a detailed explanation of how the integral approximation in BQ leads to the specific form of the aggregated loss $L^+$. The authors should also clarify the connection between the BQ weights and the conformal prediction procedure. This could involve explicitly deriving the method from a Bayesian quadrature perspective, showing how the choice of basis functions and the approximation of the integral directly lead to the specific form of $L^+$. Furthermore, the authors should discuss the convergence properties of the BQ approximation in this context and how it relates to the finite-sample guarantees of conformal prediction. This would strengthen the theoretical foundation of the proposed method and clarify its relationship to existing conformal risk control techniques. Second, the authors should provide a more compelling argument for the superiority of the HPD rule over existing methods, particularly RCPS. This could involve demonstrating a clear advantage in terms of computational efficiency or prediction set size. The authors should also provide a more detailed analysis of the trade-offs between risk control and prediction set size for the HPD rule, and compare these trade-offs to those of other methods. This could involve exploring different ways to construct the prediction set given the risk control constraint, and analyzing the resulting set sizes. The authors should also consider comparing their method to other conformal risk control methods that do not rely on randomization, as this would provide a more direct comparison of the core ideas. Third, the authors should address the limitations of the proposed method more thoroughly. This should include a detailed discussion of the exchangeability assumption and its implications for the practical applicability of the method. The authors should also discuss the challenges of applying the method to complex decision sets and propose potential solutions. This could involve exploring alternative formulations of the aggregated loss or developing new decision rules that are better suited for complex decision sets. Fourth, the authors should expand the experimental evaluation to include more complex datasets and different types of models. This would help to assess the generalizability of the findings and demonstrate the practical relevance of the proposed method. The authors should also include a detailed analysis of the computational cost of the proposed method, comparing it to other methods. This would help to assess the practical feasibility of the method for large datasets. Fifth, the authors should provide a clearer explanation of how the method can be used to construct prediction sets in regression problems. This should include a step-by-step explanation of how the method can be used to construct prediction intervals in heteroskedastic regression. Finally, the authors should improve the clarity of the paper's writing. This should include explicitly defining all notation before its first use and providing more detailed explanations of the core concepts. The authors should also ensure that the paper is consistent in its notation and that all equations are clearly explained. By addressing these suggestions, the authors can significantly strengthen the paper and make it a more valuable contribution to the field of conformal prediction.

❓ Questions

Several questions arose during my analysis of this paper. First, I am curious about the specific choice of the Dirichlet distribution for the weights in the aggregated loss. While the paper mentions that this choice is motivated by classical results on tolerance regions, I would like to understand if other choices of prior distributions would lead to similar or different results. What are the implications of using a different prior, and how would this affect the theoretical properties of the method? Second, I am interested in the practical implications of the proposed method for complex decision sets. The paper acknowledges that the method is limited to simple decision sets, but I would like to know if there are any potential approaches to extend the method to more complex decision sets. Are there any alternative formulations of the aggregated loss or decision rules that could be used in this context? Third, I would like to understand the sensitivity of the proposed method to the choice of the confidence level $\beta$. The paper mentions that the HPD rule selects a threshold based on a user-defined confidence level, but I would like to know how the choice of $\beta$ affects the resulting prediction sets and the overall performance of the method. Is there an optimal way to choose $\beta$, or does it depend on the specific application? Fourth, I am curious about the computational cost of the proposed method, particularly the HPD rule. The paper does not provide any information about the computational time required for the HPD rule, and I would like to know how it compares to other methods. Are there any ways to optimize the computational efficiency of the HPD rule, such as using parallel computing or other techniques? Finally, I would like to understand the relationship between the proposed method and other Bayesian approaches to conformal prediction. The paper focuses on a specific Bayesian approach using Dirichlet sampling, but I would like to know if other Bayesian methods could be used in this context. How would these methods compare to the proposed approach in terms of theoretical properties and practical performance? These questions are aimed at clarifying key uncertainties and methodological choices in the paper, and I believe that addressing them would significantly enhance the paper's contribution to the field.

📊 Scores

Soundness:2.25
Presentation:2.0
Contribution:1.75
Rating: 3.5

Keywords

Click the button to extract keywords

Insights

Click the button to extract insights
Version Management
Version 1
Citation Tools

📝 Cite This Paper