Let us start by reading 2 Cor 12:16-13:3.
12:16 Let it be assumed that I did not burden you. Nevertheless (you say) since I was crafty, I took you in by deceit. Did I take advantage of you through any of those whom I sent to you? 18 I urged Titus to go, and sent the brother with him. Titus did not take advantage of you, did he? Did we not conduct ourselves with the same spirit? Did we not take the same steps? 12:19 Have you been thinking all along that we have been defending ourselves before you? We are speaking in Christ before God. Everything we do, beloved, is for the sake of building you up. 12:20 For I fear that when I come, I may find you not as I wish, and that you may find me not as you wish; I fear that there may perhaps be quarreling, jealousy, anger, selfishness, slander, gossip, conceit, and disorder. 12:21 I fear that when I come again, my God may humble me before you, and that I may have to mourn over many who previously sinned and have not repented of the impurity, sexual immorality, and licentiousness that they have practiced. 13:1 This is the third time I am coming to you. "Any charge must be sustained by the evidence of two or three witnesses." 13:2 I warned those who sinned previously and all the others, and I warn them now while absent, as I did when present on my second visit, that if I come again, I will not be lenient - 13:3 since you desire proof that Christ is speaking in me.
Here Paul defends himself against criticism (12:16-18) and explains that he does so to prevent the church from falling into "quarreling, jealousy, anger, selfishness, slander, gossip, conceit, and disorder" (12:19-20). The logic of 2 Cor 12:16-20 requires that Paul is worried that criticism of himself could lead to quarreling etc.. It is not hard to imagine that if some Corinthians believers succeeded in denigrating Paul to the rest of the congregation, this could lead to symptoms of disunity such as quarreling etc..
Now, it seems to me that Paul is speaking from experience here, for 12:19, 12:21, and 13:2 all refer back to an earlier time. He must counter the slanders about himself because he knows from experience that they can lead to quarreling etc. in the Corinthian church.
Now, as commentators have pointed out, the quarreling etc. of 2 Cor 12:20 are the problems that Paul addresses in 1 Corinthians. Furthermore the sexual immorality of 2 Cor 12:21-13:3 also looms large in 1 Corinthians. Also, 1 Corinthians shows that Paul had been criticized by some Corinthian believers (See 1 Cor 4:1-5; 1 Cor 9:3 and probably 1 Cor 5:9-13 and 1 Cor 16:1-4). Some Corinthians, on hearing criticisms of Paul might well have decided to look around for another apostle and this could explain 1 Cor 1:12. It seems to me, therefore, that attempts by some to denigrate Paul had led to the problems that Paul addresses in 1 Corinthians, and that Paul says in 2 Cor 12:16-20 that he must defend himself so that the same thing does not happen again.
But who was behind these attempts to discredit Paul? I think it was those in the Corinthian church that had a sexually immoral, licentious philosophy. Here is why. Firstly, we have hints at 1 Cor 5:9-13 and 2 Cor 13:3 that they criticized Paul. Secondly, they stood to gain if Paul lost authority in Corinth. Paul disciplined a sexually immoral individual in 1 Cor 5:1-8 and threatened others in 2 Cor 12:21-13:2 and 1 Cor 4:21, but without the support of the church, he would be powerless to enforce his sexual ethics on the licentious believers. The licentious believers would be able to practice their sexual immorality with impunity if they could discredit Paul. 2 Cor 12:16-13:3 confirms that criticism of Paul increased sexual immorality in Corinth (otherwise what is the connection between 2 Cor 12:16-20 and 2 Cor 12:21-13:3? Paul is surely saying that he must defend his reputation lest the licentious continue their sexual immorality in the confidence that he will not be able to punish them.
We need not assume that the licentious attacked Paul only in relation to the issue of sexual ethics. Like a politician in an election campaign, they would seize any opportunity to smear their opponent. Thus they may, for example, have been behind the accusation that Paul intended to defraud the Corinthians (see 2 Cor 12:16-18 and probably 1 Cor 16:1-4), or the criticism that he was not an eloquent speaker, as well as the bringing of Paul to judgement (1 Cor 4:1-5).
In summary, Paul clashed with the sexually immoral in Corinth so they chose to try to undermine his authority in the church by discrediting him in the eyes of the other members of the congregation. This led to disunity.
The chief culprit seems to have been the offender of 2 Cor 2 & 7. See my earlier post here.