It more nuanced than just building a modem in this case. Qualcomm does have an extensive patent portfolio, but also their 2G-5G IP makes up a great deal of the standards. As others have stated, since it's part of the standard it has to be licensed on FRAND (Fair, reasonable and discriminatory terms).
So, they will get the (rumored to be) $7.50 per phone no matter what Apple does on the modem front, who they buy it from,etc. As to building their own modem, using Qualcomm's and others patents in the modem, they may have to pay additional $$ to utilized these