There are three main downsides of secretive state-sponsored surveillance even in a state that does not have the overt appearance of being authoritarian.

The first is that the surveillance can be used to target noncriminal activities which are nonetheless judged as undesirable by the state. For instance, you can learn all the strategies of the "wrong" (anti-surveillance) party, giving you an unfair advantage in crafting messages to defeat them. You can discover the trade secrets of companies who don't give your perfect wonderful pro-surveillance party large campaign donations and surreptitiously give the secrets to their competitors. You can target individuals to discredit them in any number of ways: you can find jilted former lovers who can be induced to say heartfelt but inaccurate things about them, you can release private information that is embarrassing but not illegal, you can gather circumstantial evidence to trigger explicit investigation (e.g. audits), and so on. Just read about COINTELPRO for some examples of this type of abuse.

The second is using the surveillance to target actual criminal activities in a highly selective or biased way. Because the surveillance is secret, it's much harder to detect whether there is actual bias or whether it's a discovery problem. For instance, if there is a 10% increase in the number of speeding tickets, it doesn't seem like anything is going on. If it just happens that those are all the most vocal anti-surveillance people, and really law enforcement has velocity data from everyone's phone all the time, that's selective enforcement. Given that we already do this without mysteriously invisible surveillance (c.f. number of black people targeted for traffic stops, as measured by the difference in racial bias for stops during the day and at night, where it's harder to tell), it's hard to believe that this isn't an ongoing problem even now and would get worse the more surveillance is allowed.

The third--assuming that the possibility of surveillance is known but the details aren't--is a massive erosion of the psychological safety around any controversial discussion. Because by design it's practically impossible to know if either of the first two problems are in effect, if you don't want to be targeted, you keep quiet and say what you're supposed to say, whether what you're supposed to say is that "transgenderism should be eliminated" or that using "they" when someone wants to be called "she" is "literally killing people"--the side doesn't matter at all, only the assessment of where power lies, and what power wants you to say. We've always been at war with Eastasia, after all.

Each of these types of problems has relatively straightforward solutions; they do, admittedly, somewhat reduce the effectiveness, but they drastically reduce the potential for abuse.

For abuse pattern one, basically "off-label use" of surveillance, the solution is to have multiple layers of intrusive surveillance-of-surveillance. People engaged in surveillance work should always expect that their surveillance is being surveilled by an independent body whose job it is to check that the use is strictly within whatever limits have been described. And those people need also to be surveilled to make sure they're doing their job, and so on. To a lesser extent, various sorts of oversight boards and organizational reviews can somewhat help, but the problem is that when everything is secret, wrongdoing simply isn't found. In the latter case, you have to go out of your way to empower and protect whistleblowers, because there is otherwise no mechanism to discover this abuse.

For abuse pattern two, the capabilities and breadth of application need to be publicly known, even though the individual instances aren't. Then already-public information on who is being charged and convicted and so on is enough to detect selective enforcement.

To solve the third problem, the control has to be in the hands of the public, not the state, at least to the extent that the issues are known in enough depth so that they can vote for or against candidates on the basis of their positions. Because the third problem is fear-based, not reality-based, the way to solve it is to expose reality to dispel the fears. Then if people want to live in a surveillance state, they can vote in people who will make it happen, and be confident that they know what they're getting into.

These aren't the only ways to address the issues; they are just examples. But the downside of authoritarian control of surveillance apparatus is very very very bad, which you seem to be not accounting for whatsoever.