I think what may separate the good from the bad is accountability, like you mentioned. A charlatan simply can’t survive forever in an authority hierarchy in which a higher authority selects him for service, regularly evaluates him according to defined standards, and holds him accountable to these standards.