The desired He3 reactions may not directly produce any neutrons but in the environment created for fusion it would be nearly impossible to prevent stray interactions from kicking out neutrons. Neutron embrittlement is one of the top problems I've read about in regards to the viability of fusion.
It's actually far less of an issue for fission reactors since they use water as a moderator. This protects critical components such as the reactor vessel from neutrons. The only thing neutrons regularly interact with is the fuel assembly which gets replaced every ~5 years or so.
If water stops neutrons completely, then protection designed for ITER will also do so. I have read that neutron embrittlement of reactor walls limits safe operation time of a fission reactor. Reactor contains high pressure hot water (for steam turbine) and all fuel. If it breaks, result is Tsernobyl/Fukushima again. If fusion reactor breaks, small amount of mostly non radioactive hydrogen escapes. I don't know how much fuel fusion reactor has, but it cannot be much. Wendelstein 7-x stellarator used 1 mg of helium in its first plasma test. So both suffer from neutron embrittlement. It is safety issue for fission.