Good, so your tree is definitely Olea europaea. Do try thinning the fruit and see if it will enlarge the olives. The effects of thinning are partly tree-wide due to a higher leaf/fruit ratio, but there is also an effect that is local to each branch just because the fruit are not crowded. You could try thinning some branches and not others to see if you notice a difference.
One other caveat is that fruit size doesn't always signify quality. There are some very choice olive cultivars ('Arbequina'. 'Gaeta', etc.) that are known for producing small, intensely flavoured fruit. I don't know if they are thinned or not.
As a side note, I've learned that some commercial olives in California are 'hormonally thinned' with NAA (1-Naphthaleneacetic acid), a synthetic auxin. I'm not sure if this feasible or recommended for a home grower. I'm also not sure if this practice is allowed with organic farming. I wouldn't think so.