Lag when swiping away recent apps

I'm running Lollipop 5.1.1 and whenever I swipe away recent apps in the task switcher such as Chrome or Play Store, there is a lag/stutter before the app leaves the screen. It swipes away and then pauses for a second before leaving. Is there any way to fix this.