There is a sign on the top that says there are 4 items in the review queue for me but when I look at the review queue there is none.
This issue has been going on for a few weeks.
Yes. This is a known consequence of the design of this feature, and the StackExchange team has indicated that they do not plan to fix it. The issue is that they cache this number, and the cache gets out of date. Also, they compute the total number of reviews, rather than the number that you personally can perform, because they don't want to compute it separately for each user, which can also cause the number to be inflated. Computing the correct number for each user would be too computationally expensive, apparently.
So, in summary: known defect, no plan to fix it.