You can have the Photos app at the top of the memory.
Tap Feature Less, then depending on the memory, tap one of the following:
Feature a Person Less
Feature This Place Less
Feature This Holiday Less
Feature This Day Less
Feature These Days Less
If you choose Feature a Person Less, select the person you want to show fewer photos of, then tap Next. Choose Feature This Person Less or Never Feature This Person, then tap Confirm.
Tip: To turn off all memories that feature holiday events in your home country or region, go to Settings > Photos, then turn off Show Holiday Events.
Note: If you choose to feature a person less often, but then want to see that person again, you need to reset your Memories settings. Go to Settings > Photos, tap Reset Suggested Memories, then tap Reset to confirm.