The Northern Ireland Office confirmed that Mr Brokenshire has resigned, but would not confirm reports that his decision was based on health grounds.
Mr Brokenshire said the parties in Northern Ireland have gotten over bigger issues in the past.
Karen Bradley, 47, was given the job, moving from her role as head of the government's Department for Digital, Culture, Media and Sport, where she is best known for adopting a tough stance over Rupert Murdoch's bid to buy all of broadcaster Sky.
Brokenshire was criticised for continually resetting deadlines in talks to restore the region's power-sharing government.
"When the previously unheard of James Brokenshire was appointed Northern Ireland secretary of state, journalists at his local paper in south London were asked about their MP".
He wrote: "In the last few days I have been told that I have a small lesion in my right lung which needs to be removed".More news: DeWanda Wise joins cast of 'Captain Marvel' movie
The 50-year-old father-of-three stepped down this morning, citing a lung condition as the reason.
Brokenshire, who has led government efforts to restore the devolved Northern Irish executive after power sharing collapsed previous year, was stepping down ahead of major surgery in coming weeks, his spokesman said.
"A key part of my role will be to help build a Northern Ireland that is fit for the future and works for everyone", she said.
James Brokenshire was Northern Ireland Secretary from July 2016 to January 2018Who is James Brokenshire?
She made only limited changes among her senior ministers following the election in June having seen her position badly weakened by the loss of her overall majority in the Commons. "Business-friendly" was one description, however, beyond that they found it hard to characterise the former immigration minister.
It is thought that Mrs May will take the opportunity to bring forward some more junior ministers, with Immigration Minister Brandon Lewis and Justice Minister Dominic Raab among those tipped for promotion.