This is a question many yogis around the world struggle with. It is also arguably one of the main questions that keep people from trying out Yoga, another being the myth that Yoga isn’t intense.
Hatha Yoga is a practice where we work with our body and breath to calm and quieten our mind, please keep in mind that the word yoga is used here to refer to traditional Hatha yoga practice. To answer the question, let us start by going to the dictionary definition of the word religion-
Religion is defined as the belief in a god or in a group of gods / an organized system of beliefs, ceremonies, and rules, used to worship a god or a group of gods.
The key word here is belief which is indicative of the nature of most religions where we are required to hold faith in a supernatural force which one may have never experienced. Because Yogic practices originated in a country which was predominantly Hindu, people associate yoga to Hinduism. Hinduism itself can be defined at best as a polytheistic religion which is a melting pot of a multitude of beliefs and rituals. Swami Vivekananda famously quoted that ancient India was a land of many religions, each person having his own – meaning that each person had his own path to follow. Religious rituals and practices differ a lot within India. For e.g. a Hindu from the south of India will not recognise most of the rituals and beliefs of someone living in the northern parts of India when praying to the same deity. So if we are to understand Yoga (Hatha Yoga) in the context of Hinduism we can say that it was just one of the numerous streams of practices aimed at spiritual growth and self-actualisation, which one could say is the end goal of most spiritual paths that came to evolve under the umbrella of Hinduism.
Now going back to its definition, religion by its nature requires you to believe in something which you have not already experienced, whereas yoga is all about the experience of life itself. Yoga in its core does not expect us to believe anything blindly but only to do our practice diligently. While one has to have a certain level of trust in the practice there are no rituals or practices which are expected to be done without understanding the purpose of it. Hatha Yoga can be considered as a learning process where we start from body awareness & control through various asanas, then progress to breath awareness & control through pranayama and then start becoming aware of the mind and eventually master it and go beyond. The key word here is awareness, which means we are not expected to believe that an asana gives a particular effect but we should become aware of our body through the asana. This is a very personal experience, while you may be learning as a group initially, with practice you start becoming more aware of yourself in body, breath and eventually mind.
It is better to avoid chanting than continue chanting with doubts in your mind about it because instead of calming the mind we are agitating it further.
Once we remove belief from the equation, one of the major reasons for yoga being considered religious is the various Hindu rituals associated with it starting from chanting OM, lighting a lamp or incense to the way many yoga practitioners dress up. While some of these like chanting OM or lighting incense can provide a certain ambience to the practice, it is not the core of the practice. If you do not want to chant OM, it does not make your practice meaningless or weak. OM chant is one of the ways to help regulate our mind and keep it calm and still. It is better to avoid chanting than continue chanting with doubts in your mind about it because instead of calming the mind we are agitating it further. This can be extended to any of the so called rituals associated with the practice. Having said that, let me also add that OM chanting the way it is used for meditation practice has nothing to do with Hindu religious beliefs. It is not a prayer to any deity but considered as a Mantra to raise your own self-awareness. Some call it the primal sound, but you can take it mean anything that will bring your awareness back to yourself. The fact that it is a Sanskrit word adds to this notion of connection to Hinduism, but keep in mind that Sanskrit is a language that was used in ancient India. It is like saying that if I teach a class in English I am practising Christian yoga.
In fact, Hatha yoga practice can be considered more atheistic from the point of view of religion in the traditional sense. In Hatha yoga practice we are not trying to reach out to a supernatural force that resides outside, but we are trying to become more self-aware and go deeper into ourselves. These associations to rituals stems from the fact that many instructors while being good at the physical practice, do not pay much attention to the spiritual aspects of the practice. In the effort to appear like an authentic teacher or studio, many add these rituals or symbols and thereby alienate those who want to practice but associate these to religion.
I would conclude by saying to those who want to start Yoga practice but consider it religious, make the effort to find an instructor who focusses on the practical aspects and do not bother to make an elaborate spectacle out of it.