US consider Russia as their only rival since the Cold war days. Even After Soviet Union disintegration, US can't get rid off Russia from their Nightmares. Now coming back to the S-400 deal and Indo-US relations,
Modern world partnerships are mainly based on monetary benefits. There is no emotion involved in the equation. As the largest importer of defense equipments in the world, India assert a certain level of influence in such transactions. Losing such a deal is a huge blow when it comes to US reputation of arm trades.
US has already imposed huge sanctions on Russia post the presidential election. They suspect Putin involvement which resulted in closing down of embassies and officers, agents being sent back to their natives.
In such a commotion, US finds one of it's closest Ally India doing a hand to hand deal for S-400s with Russia. Deals were already in progress before the sanctions. India can't back off at this juncture since Russia is one of it's oldest defense partner and have a history of coming for aid in the past during dreadful times when even US showed their backs.
As far as India is concerned, it took a right decision since that portray the commitment the nation has showed to it's defense partners. In the same time, India agreed to buy Apache helicopters from US at a huge price level thus maintaining the balance.Politics played well.
Both Putin and Modi showcases a warm relationship in international summits and their closeness even transpired to such deals.
In our opinion, India should currently concentrate on improving relationship with BRICS partners rather than running behind super powers since neighborhood matters a lot than a distant relative.