ALBANY, N.Y. (AP) — A Manhattan federal judge let New York proceed Wednesday with its plans to move June congressional and state Senate primaries to August, but asked the state Board of Elections to seek approval from an Albany federal judge. An independent court expert is crafting a new set of maps by May 20 under the oversight of a state judge in Steuben County, who ordered the delay of June 28 congressional and state Senate primaries to August 23.

U.S. District Judge Lewis Kaplan rejected the application for a temporary restraining order from a lawsuit filed by New York voters who wanted to keep the possibility of June 28 primaries open. The voters, represented by Democratic attorney Marc Elias, said it’s too late to draft new maps in time for a June primary even though state appeals judges found the maps were unconstitutional.

The voters in their lawsuit claimed that New York has to hold their primary in June because of a 2012 court order by Albany-based U.S. District Judge Gary Sharpe of the Northern District of New York. But Kaplan said that New York can easily ask for permission to delay the date to August from Sharpe, who’s previously approved New York’s requests to tweak its political calendar.

The 2012 court order says New York can choose a different non-presidential federal primary date as long as it complies with voting rights law. “Everybody agrees on what the goal is,” Kaplan said. “The question is: how to make it happen?”

New York’s gubernatorial and Assembly primaries are still set to take place on June 28. State Board of Elections attorney Brian Quail told Kaplan the board would formally ask Sharpe Thursday to delay the primary to August.

Christina Ford, an attorney representing the voters, argued that New York lacks time to craft new maps and meet all voting rights deadlines by August. She asked the judge to keep the rejected maps in place and require New York to proceed with a June primary for now.

But Kaplan said the attorneys failed to persuade him to trigger a process that could lead to voters electing representatives in districts that the state court has deemed unconstitutional. “What are candidates supposed to do?” he asked. “What are voters supposed to do? What are all the people who are concerned with elections supposed to do? I would be hard-pressed to imagine a scenario that would cast greater disrepute on the rationality, the fairness, the consistency of the holding of elections in this great country.”

Court of Appeals judges ruled the Congressional map unconstitutionally gerrymandered, and found that lawmakers lacked the authority to pass state Senate and congressional maps once a bipartisan redistricting commission failed to come to a consensus. They said they didn’t toss the Assembly maps because a lawsuit filed by GOP voters didn’t specifically challenge them. On Wednesday, former Democratic candidate Gary Greenberg filed a lawsuit asking a state judge in Steuben County to toss and order new Assembly maps and delay Assembly races until August 23 as well.