I went with carbon wheels on my XC bike because I was competing seriously at the time and wanted every advantage I could get. Super light carbon wheels did help. Though being a light person in could (and did for a while) get away with something light like Stan's Crest and save money. Besides the bike eventually getting stolen, I feel like it was worth it.
I also went with heavy carbon on my Enduro bike after destroying something like 4-5 Flow's in a year. While the DT held up better (it is a spare in the van), I was having to true it constantly. I built up some Race Face Arc Carbon with 32 strong spokes (I broke the original 28 spoke carbon on the Enduro) and two years later I rarely have to adjust spokes, let alone break a wheel. I did have to go with inserts as the lack of broken wheels meant I was getting more pinch flats, as the stronger carbon wheels let me push the limits more (also figured I would test the two year warranty, never did). Weight wasn't a concern, it was strength. If these fail my plan was to go with We Are One downhill wheels.
I'm not going back to aluminum, except on a casual road bike. The gravel bike has nice aluminum wheels, but I'll eventually replace those too. Roadie has nice, but old carbon and they have been flawless for 30,000 miles.
I felt no difference in stiffness and all that. I did go wider on the front, but I am pretty good at leaning the bike over onto the cornering lugs and don't think going wider helped me. I went from 29mm fr/rr (Flow) to 29/25 (Flow/DT to 36/31 (carbon) on the E29.