Honestly if such practice is so common, instead of average they should use some other measure as consensus:
1) Just use the mean. This way outlier from both side would not be able the manipulate the true expectation as much unless they become the overwhelming majority.
2) Weighted average, with weight calculated based on past performance. The firms that consistently miss the reality by big margin would be assigned a lower weight so that their number do not carry as much impact to the average.
Alternatively might as well find a simple linear ML model and train on past data, just that you may need to do a weight re-normalization if you are missing inputs from some of the previous firms.
If someone can supply me the past estimates from different firms I can try and train a simple linear model for best estimate prediction.