Materials I will never work with
Lead has no safe exposure levels and causes brain damage, so I avoid it as much as possible. Unfortunately, it's the primary plasticizer for PVC and cord coatings, a pigment in old paints, and the electrodes in car batteries. It's also a significant component of most highly-machinable steel, aluminum, and brass alloys, but there are Lead-free machinable metal alloys.
Brass often contains lead and is the typical alloy used for keys, door knobs, and pens; all meant to be handled.
Mercury is a neurotoxin and evaporates into the air -- the vapors are harmful and it's extremely difficult to contain. This means that compact fluorescent bulbs and fluorescent fixtures are problematic.
Styrene causes cancer and is neurotoxic. 3D printing ABS plastic and melting styrofoam release styrene vapors.
Americium or any other radiation source. Some smoke detectors use Americium to detect particles.
Per-fluorinated compounds like PFOA, PFAS, or PTFE (Teflon). These are ridiculously abundant in products and even are even applied to disposable food containers. The waterproofing industry seems to be pivoting to "short-chain" versions of these that supposedly break down more readily in the environment. Another name for these is DWR, or durable water repellant.
Leather is produced from a dead animal. I wouldn't kill an animal myself, so I shouldn't pay for it to be done on my behalf. Also, tanning compounds can be pretty nasty.
Unsustainable wood comes from old-growth or otherwise poorly managed forests. Wood can be harvested sustainably, but often at a higher cost.