Some people can bring positive to your life while other people can bring you negativity. This is a very individual thing since what is negative to you can be positive to someone else.
First, remember that everyone is doing the best they can when you take into account the life that they have lived and the many lives they have lived before that. In fact, if you were literally them, you would have made the same decisions they made every time. This is important because to do the next steps, you need to come from a place of non-judgment. This will allow you to prevent feeling guilty which then prevents karma.
Invite the person who you are unsure about to join you at a restaurant or cafe. You can invite them to your house if you choose but it can be preferable to only invite people you have already decided are good for you into your personal space. Released negative energy can stick, making your whole house feel heavy until it is cleared.
Arrive at the restaurant or cafe before the other person. Have a seat and try to assess how you are feeling before they arrive. After the other person leaves, ask yourself if you are feeling better or worse. Try doing this a couple of times and keep track of your assessment.
If you consistently feel worse after meeting with them than you did before, consider removing them from your life.
If the person is currently going through a major personal issue, you may want to give them some leeway. However, if they are always going through a major issue, consider removing them from your life.
You are not being a bad person for removing them. In life, we walk a path and sometimes meet people who are on the same path. At some point, the path can split and one of the two people have to either be taken off their own life journey or they have to both go in their individual directions.
Finally, check your expectations. Every time someone leaves your life, someone else comes in to fill their role. Make sure your expectations of the next person are not dictated by the negative traits of the previous person. Imagine what the new person would be like. This will help you manifest what you want instead of what hurt you.