You were in top now. Your large target audience doesn’t affected by inaccessibility of the site. But, it might affect other people, minority users. Accessibility is useful for older people too. Accessible site is easy to use for people accidentally lose their visions, hearings, and hands ability. Your target audience may changes because of aging process and a lifecycle.
This unessential side is not going to push you to apply all recommendation. You still have a rights to don’t care about it.