Huggingface trainer has parameter compute_metrics. This function however receives only predictions and labels as its input:
def compute_metric(pred_label):
pred, label = pred_label
return my_metric(pred, label)
This is sufficient for computing say Acuraccy, but unsuitable for more complex metrics like NDCG, since NDCG also requires information about group_id.
The typical example is information retrieval, where you benchmark set of queries (group_ids) each with different order of preferred results.
Is there a way to access information about group_ids in compute_metrics?
Furthermore, is there a way to access eval dataset name inside compute_metrics?
