If a card refers to "banished cards," it includes all banished cards regardless of which player banished them. The rulebook states:

“A few end-game bonuses refer to the pile of cards that were banished during the game. It doesn’t matter which player banished these cards.”
(page 6)

This means the term "banished cards" always refers to the entire pile of banished cards, not just those banished by a specific player.

Source: page 6

Source: Core Rulebook (pp. 1, 4, 5, 6)