Hi and welcome
valedevento - although I do have to admit you have me slightly confused as your location says Belgium … yet here you are in Portugal
but either way in answer to your question.
What you experienced is not something that can be changed - for the simple reason - that its actually quite normal - as - apart from a few fairly rare cultivars - the majority of olive trees are what are known as alternate bearing - meaning - they have a natural tendency to bear fruit one year and little to none the next. So as you did get some - even though they were small - you should consider yourself lucky - especially as - more often than not ... its none on the trees that are having their non yield bearing year.
Regarding fertilizer - as too much fertilizer and especially if the nitrogen content is high - can inhibit fruit set - olive trees generally do best with only a light dressing of fertilizer once a year and preferably with a potassium rich fertilizer rather than one with a high nitrogen content.
Also - as olive trees are best suited to hot arid desert like climates with an extremely long dry season - meaning around 6 to 8 months or more without rain - they are best kept on the very, very dry side - especially as moisture is an olive tree's worst enemy - as not only does too much water quickly lead to various fungal diseases and root rot but also an untimely death.
In fact - normal practice with olive trees is to not water them at all - but let nature do the watering - as so long as they get some rain - even if only a small amount - between October and December - that is more than adequate to ensure a healthy tree and an abundant harvest.
Oops nearly forgot - its also normal practice when starting an olive grove to stagger the planting schedule - by planting half the trees one year and the other half the second year - as by doing that - because they are alternate bearers - it ensures that you will have a crop every year.
