The comment was made in a private meeting with the historian Nicholas Murray Butler. Butler recounted the anecdote, and the context in which it was made in his book Across the busy years: recollections and reflections, Volume 1, published in 1935 (more than a decade after Warren Harding’s death).

Butler prefaced the story with some opinions about Harding’s character:

Harding was one of the kindest men who ever lived, but he was without any serious qualifications for the Presidency. He had a good mind, but made little use of it. He had no wide or accurate knowledge of public questions or of the foundations in history, economics and public law on which those questions rest. He was good-natured, lazy and weak when pressure was put on him by a stronger will than his own, which happened to be that of a friend. He would not have consciously done a wrong act in his great office but he had neither the intellect nor the character to prevent himself from being made use of for unworthy purposes by unworthy men who loudly professed their personal and political friendship.

He then went on to recount an example where he remembered:

… coming into the White House offices about six o’clock in the evening and being urged by Mr. Christian, the President’s secretary, to persuade the President to go to the White House and rest before dinner. Christian added that the President was very tired and was at that moment sitting before a huge pile of letters in his private office.

Butler recalled that he went to President Harding and found him

… precisely as his secretary described, but with a look of extreme dejection on his face. In reply to my urging that he come over to the White House and lie down before dinner, he said with a weary groan that he must go through this pile of letters which he had not as yet found time to examine. Having known the way in which McKinley and Theodore Roosevelt did business in that office I said to Harding, “Do you mind my looking at some of them?” He replied, “No, look at any of them you please.” Taking the first two or three letters off the top of the pile which must have contained not fewer than one hundred, I glanced at them hastily and said, “Oh, come on, Mr. President, this is ridiculous. Even in my office they do not burden me with reading or answering letters like these.” I shall never forget Harding’s answer, for while it was very pathetic, it did him great credit. These were his words: “I suppose so, but I am not fit for this office and should never have been here.” Other Presidents might truthfully have said the same thing, but it may be doubted whether any of them has been or will be frank enough to make the confession which broke from poor Harding’s lips that evening.

Nicholas Murray Butler – Across the busy years: recollections and reflections, Volume 1. pp410-411 (my emphasis)

Of course, the line has been taken (generally without context, and often without citation) by those who wish to portray Harding as “the worst ever American President” to prove their point by using Harding’s own words to convict him. In context, however, the admission, and Butler’s following remark, make the claim far less damning.

The exact date of the meeting is not made explicit in Butler’s book. The preceding letter quoted in the book dated to 1921, while the anecdote that follows dates to 1923 (Harding’s Presidency lasted from 4 March 1921 to 2 August 1923). So the best we can say is that the meeting did indeed occur while Harding was in office, but we cannot narrow the date down further.