Metrics

The classic SGG metrics of

R@K

and

mR@K

are used to benchmark PSG models. Notice that PSG grounds objects with segmentation, a successful recall requires both subject and object to have mask-based IOU larger than 0.5 compared to their ground-truth counterparts, with the correct classification on every position in the S-V-O triplet. The panoptic segmentation evaluation protocol

PQ

can be also considered as an auxiliary metric.

Participation

The PSG benchmark is live on Paper-with-Code. You can benchmark your PSG models and update the results in the benchmark. Alternatively, you can directly send your results and associated method descriptions to jingkang001@e.ntu.edu.sg. We will update your methods manually into our benchmark.

Leaderboard

Method	Backbone	#Epoch	R/mR@20	R/mR@50	R/mR@100	Date
IMP	ResNet50	12	16.5 / 6.52	18.2 / 7.05	18.6 / 7.23	CVPR'17
MOTIFS	ResNet50	12	20.0 / 9.10	21.7 / 9.57	22.0 / 9.69	CVPR'18
VCTREE	ResNet50	12	20.6 / 9.70	22.1 / 10.2	22.5 / 10.2	CVPR'19
GPSNet	ResNet50	12	17.8 / 7.03	19.6 / 7.49	20.1 / 7.67	CVPR'20
PSGTR	ResNet50	60	28.4 / 16.6	34.4 / 20.8	36.3 / 22.1	ECCV'22
PSGFormer	ResNet50	60	18.0 / 14.8	19.6 / 17.0	20.1 / 17.6	ECCV'22