Numerous federal prosecutors who were nominated by Mr Obama had already left their positions before the request, but some four dozen had stayed on in the first weeks of the Trump administration.
"I did not resign", Bharara tweeted Saturday.
Bharara, the Manhattan federal prosecutor who was asked by President Trump to remain in his post shortly after the election, was sacked after he refused an order to submit his resignation, The New York Times reported. He added that serving as the federal government's prosecutor in Manhattan "will forever be the greatest honor of my professional life".
President Barack Obama nominated Williams to the post on March 29, 2012, and he was confirmed by the U.S. Senate on August 2, 2012, according to his office website.
In frequent public appearances, Bharara has decried public corruption after successfully prosecuting over a dozen state lawmakers, Democrats and Republicans alike.
He went on to say in the tweet that being the US Attorney in the Southern District of NY will "forever be the greatest honor of my professional life". Career attorneys are to carry on the work until new USA attorneys are put in place, the Justice Department said.More news: Oil slips on concerns over Chinese demand, Russian output
Montana's U.S. Attorney Mike Cotter said he received a phone call from a Justice Department official telling him "the president has directed this".
Bharara has overseen the Southern District of New York, which includes Manhattan, the Bronx, Westchester and several counties north of New York City, since 2009. "The Attorney General has now asked the remaining 46 presidentially appointed USA attorneys to tender their resignations in order to ensure a uniform transition".
One U.S. attorney appointed by President George W. Bush - Rod Rosenstein of Maryland - remained on the job for the entire Obama administration and is the current nominee for deputy attorney general.
"While it's true that presidents from both parties made their own choices for US attorney positions across the country, they have always done so in an orderly fashion that doesn't put ongoing investigations at risk", the New York Democrat said. "This saddens me because many of these people are great public servants and now they are being asked to leave".
"I think it's very unprofessional and I'm very disappointed", he said.
"I gotta write that [resignation] letter".