Arguably everything that's "physical" but not "fundamental" can be argued to be idealistic. If heat doesn't exist then neither do chairs - they are both measures of groups of particles, and not particles themselves.
Physicalism doesn't actually need to be opposed to idealism, as long as the implementation of those ideas happens because of physical things. Evolution is an idea, and it's implemented in our universe by physical things. Algorithms are idealistic, and they're implemented in our universe by physical things.
Physicalism isn't the denial of all abstract ideas, it's the statement that, if those abstract ideas "exist" in our universe, they exist in the context of physical things - they are implemented within physics, by the interactions of physical things.