"I've recently secured permission for an old site that is VERY heavy mineralized sandy loom..."... Congrats on gaining access to an old site. Now, since you're using the MXT, what is the Ground Phase Reading you get when balanced in the Prospecting mode, since you say it is VERY highly mineralized?
".. with an incredible amount of iron. Mostly old square nails."... All this tells me is that if you're not wanting to recover all the iron to clean the site, then you should be hunting with a smaller coil and only enough discrimination to reject (barely) those shallow/surface iron nails. Also, you should be using slow, controlled sweeps.
"Have been there twice trying to figure out my best performance options for my MXT but am finding this site most difficult."... How is it more difficult that other sites you hunt? If you've mostly coin hunted in-town locations and are not used to hunting some plowed fields, etc., then it will take a little time to get used to it. Also, I encourage you to just knock out most nails and recover ALL good-sounding hits as well as the 'IFFY' responses. DO NOT rely on the visual TID too much!
"Brought an experienced buddy out yesterday for three hrs. and he was able to pull out an eagle penny, an 1882 Can. 5 cent piece (same as your half-dimes) and an 1859 Can. large cent. He uses an XLT with a 5.3 concentric coil."... EXPERIENCED .. right there explains why he most likely had some success. He was using the best smaller coil for the XLT (I have the 5.3 BullsEye mounted on my own XLT and that's what I'll be using most of today), and a couple of questions come up. Did you compare your MXT over his located coins BEFORE they were recovered? If so, how did it respond? Did he dig only those targets that locked on 100% reliable, OR, is he a savvy, experienced detectorist who knows to go after those 'IFFY', questionable target responses, too?
"I tried both my stock coil and 4x6 mini-shooter."... If it's really trashy I'd use a 5.3 Eclipse (with mt M6) or in your case, the small elliptical coil.
"My buddy commented that this site contained the toughest soil conditions he's ever hunted."... Okay, but still he seems to have been able to defeat the conditions challenges. That, or you were spending too much time trying to figure out your unit then actually detecting, OR ... quite possibly ... he just happened to put his coil over the targets.
"The soil's being turned over next week so I thought I'd ask for some advice as I'm hoping to cash in on some goodies here."... Good luck next time you're there. Remember that some coins and targets will be turned under and some possibly turned up. Also, most likely the targets will be in odd positions and not nicely 'flat' to the coil, therefore you're likely to get less than perfect audio and visual ID info.
"I generally have decent success but I'm having trouble at this spot."... Sorry to hear that. Just relax and work at it as best you can.
"Does it advantage me to minimize my gain setting a bit?"... Possibly, but what is "a bit?" With the MXT's I owned (4 of them) I NEVER hunted with my Gain below 8, and it was only that low on maybe four or five occasions. I always try to run any detector as high as possible, 'max' if able, and reduce it just to the point I regain stability. The lowest with them, or with my present Matrix M6, is at the factory ▼ marker.
"I generally hunt in relic mode with trigger switch forward. I run my gain close to max most of the time. I'm wondering if might not be better off toning my machine down a bit here."... I would suggest using all the Gain (Sensitivity) you can in the Coin & Jewelry mode with a smaller coil, and the Discrimination set to just barely knock out those surface nails. Do NOT rely on TID. Dig the beeps!
Also, take along your IDX Pro and if your hunting buddy will let you, borrow his 5.3 BullsEye coil to use on the IDX Pro. My Classic III SL & IDX Pro have Bill Crabtree's modifications and I am able to get the best GB for any site and coil in use, but I prefer my Classic's to the MXT/M6 when hunting the trashier, iron-infested sites.
Oh, another thing. With the MXT/M6 I prefer to get a good GB setting, slightly positive for the ground, and then LOCK the toggle to take it out of the tracking mode. IN many environments with a lot of iron trash, I get better results with AutoTrac
Again, I can't thank you enough for your time and expertise. Your reputation is well known in the hobby, even to beginners like me !