First start the syllabub. Put the sweet sherry or marsala in a bowl with the brandy (if using) the lemon zest and caster sugar. Stir until the sugar has dissolved.
Leave this to stand for at least the time it takes to assemble the trifle, but for several hours if you are able to prepare ahead.
To assemble the trifle, first spread half the trifle sponges or fingers with the raspberry jam and arrange them in the base of a trifle bowl. Pour over the sherry and top with the amaretti biscuits, followed by the raspberries.
Spoon the custard over the raspberries, then cover and leave to chill.
Now finish the syllabub. Whip the cream until it reaches the soft peak stage, then stir in the sherry mixture and whisk again to combine. Spoon this over the trifle, then sprinkle over the toasted almonds and some extra lemon zest.