My first question is this: What business reason prompted you to consider agile? Is that business reason still valid? If it is, you want to consider debugging your project.
What is not working? Are iterations not working? Why?
Are requirements as user stories not working? I would determine how to make them user stories or smaller to make them work.
Is the cultural change not working? Ah, that's a problem. In that case, I would use a kanban board. Maybe even kanban inside iteraitons.
Are you doing retrospectives? If not, you need to. You need to understand what is going on and how things got this way.
I could go on, but I'll stop.
You have many choices for lifecycles other than waterfall or agile. You have iterative, incremental, or iterative and incremental. Agile is one form of iterative and incremental. Waterfall is one form of a serial lifecycle.
If you return to the business reason that prompted you to think of why you wanted to leave waterfall and transition to agile, maybe you will have an answer.
Of course you can go back. Maybe you want to go forward, differently.