One of the more interesting aspects of writing as a craft is that there really isn’t anything that’s utterly forbidden. Just about anything can be justified in some circumstances – it’s identifying when it’s the right time to break the usual rules that’s the challenge.
This is why an engaging voice and vivid, likable characters can carry a book with serious technical problems. It’s also why being “transgressive” for the sake of transgressing rarely works well in fiction.
To start with, no matter what social norm you think you’re transgressing against, it’s guaranteed that someone, somewhere in the nether hells of fanfiction has already done that. And if you don’t have a solid grasp of craft, someone’s done it better. Hell, even if you do have a solid grasp of craft, someone’s transgressed more impressively than you have.
That’s one of a very few damn near absolute rules I’ve come across (most of them involve hard sciences). No matter how good you are, there is always someone better. And Murphy’s Law suggests that you will encounter that someone better in the most humiliating way imaginable (or not imaginable, since imagination has limits. Life tends to ignore them). It helps to remember this. (Also, if you plan to write smut, it helps a lot to remember that biology imposes certain firm limits on what can be done. Do please research these, since it really doesn’t help your cause to have readers go into fits of hysterical giggling when they try to figure out just how appendage A is going to get into orifice B without breaking something or detaching from its owner).
So yes. There is always someone better. There is always someone smarter than you, someone who knows the field better than you do. Even if you are the undisputed master of your field, you aren’t the master of any other field, and sooner or later, someone will rise to take your place in your field.
This is called life. It doesn’t usually find its way into books for the simple reason that stories tend to focus on the extremes, and the person who has been on top and is facing the rising star is, if they show up at all, usually not the protagonist (that would in most cases be the rising star). But again, even if you want to transgress against this, someone somewhere has done it.
That doesn’t mean you can’t put your own spin on it: there are a very small number of core plots (precisely how many varies depending on who you ask), and a finite number of ways to express them. This is why how you do things matters. It’s the difference between politely asking someone if they could introduce you to their parents because you know a priest who would be delighted to marry them, and shouting “You bastard!” (Of course, you can use the nature of your characters to make a point – when the normally polite person is driven to shouting, “You bastard!”, you know they’re overloaded).
But when – or if – you choose to follow the dark path of Literary Fiction and be transgressive and gender-fluid and all those other buzzwords, you’re going to find that not only have other people done nihilistic despair better than you have, nobody (or almost nobody) wants to read that. This is not because you’ve written it badly, or at least, not necessarily because you’ve written it badly.
It’s because reading to be depressed is something few people choose to do. We want some hope in our diet of misery and despair, because that’s as much a part of being human as anything else. We hope, and we look for something better, and we take our inspiration from those who overcome the horrible. As much as we might sympathize with the ones that can’t manage to escape the crab pot, we humans need to believe that there’s something to aspire to. Something worth fighting for, and yes, if necessary, dying for.
This is “worthy” works deemed by the literary establishment mostly sell like shit.
Me, I prefer to write – and read – things that leave me feeling hopeful and maybe even inspired.