The ensign had been flown over No 10 since 2022 to celebrate Middlesex Day on 16 May.